【iOSDC2024 #LT】6. Extra support for visionOS of PianoUI
This article summarizes the implementation of animations in the presentation slides for the LT session, A Digital Piano Came to My House, So I Made an App to Connect with It!, presented at iOSDC2024.
The previous article was “5. Implementing AutoScrollView”.
This article focuses on making the app that connects to the digital piano work on visionOS. However, merely building and running the app isn’t enough; it’s also essential to consider usability and practical use. Since the UX of visionOS significantly differs from that of iOS and iPadOS, an app that works well on iOS/iPadOS might not function properly or have poor usability on visionOS. In this article, I will summarize the adjustments made for visionOS compatibility.
Additional Adjustments for visionOS
Hover Effects During Focus
In visionOS, you can click by focusing your gaze and then pinching with your thumb and index finger. This app originally allowed you to tap the piano keys to play sounds on the digital piano, but since the UI used onTapGesture for the keys, the hover effect wasn’t automatically applied, making it unclear where the focus was on the UI (a similar issue occurs with mouse cursor support on iPadOS).
While using Button might work, in visionOS, the system automatically changes the layout to a button UI, which wasn’t ideal for the piano keys (it may be possible to maintain the piano key UI with Button, but I gave up as it wasn’t easy to implement immediately).
To address this, I added a custom hover effect to indicate focus to the user by applying an effect when hovering. I referred to the WWDC2024 session Create custom hover effects in visionOS for this implementation (This app is currently only available on visionOS 2 or later due to this). Below is the code I implemented:
.hoverEffect { effect, isActive, _ in
effect.opacity(isActive ? 0.9 : 1)
}By using the hoverEffect, you can apply a specified effect when hovering. In the code above, the opacity is set to 0.9 when the key is hovered. Initially, I applied a scale effect similar to other buttons, but since the piano keys are tightly aligned horizontally, scaling made them difficult to see, so I chose to adjust opacity instead.
Adjusting the Gesture for Playing Keys
Similar to hoverEffect, visionOS operates by focusing and clicking, but the DragGesture initially set for the piano keys in iOS/iPadOS didn’t work correctly in visionOS, occasionally failing to send the event when the key was released.
https://ulog.sugiy.com/iosdc-swift-piano-layout-piano-ui/Therefore, I switched to using onTapGesture instead of DragGesture in visionOS. However, onTapGesture alone doesn’t capture the end of the tap gesture, so I added a delay before sending the event that the key was released.
Adjusting the Offset of the Keys
In visionOS, it’s easier to tap the keys when they’re slightly moved forward on the Z-axis, so I adjusted the keys to appear closer on the Z-axis.
.offset(z: 20)However, if you set this value too high, the keys might be displayed in front of Alerts, making them invisible, so you need to adjust the position carefully.
Conclusion
This article summarized the adjustments made for visionOS compatibility in PianoUI. The UX of visionOS differs greatly from that of iOS and macOS, and the available APIs are different, so individual adjustments are often necessary. Since I’m not yet fully accustomed to using VisionPro, figuring out the best way to support visionOS will require ongoing trial and error.
The next article is the “Summary”.
