Supporting screen readers is an important function for UI in commercial apps. It’s far from the entire story for accessibility but an important base functionality for building an app that can be used by everyone.
Supporting screen readers can also be technically challenging. Ideally you would want your UI framework to have a level of built-in support, meaning all UI elements are discoverable by screen readers and provide some context about their default role.
Supporting screen readers also means making an app keyboard accessible. It’s already possible to patch this functionality into any vvvv app but again if a default keyboard solution was built into the UI library this would be better.
A stretch goal would be that a solution developed now has the potential to be cross-platform. (Given vvvv doesn’t have a cross-platform UI at all yet this is quite a stretch. )
This isn’t about making the vvvv editor accessible, this is about UI that can be used by end-users in exported and commercially distributed apps.
Status today
UI elements made with ImGui or Elementa are not discoverable by screen readers.
VL.CEF web content rendered in skia or stride is not discoverable by screen readers
Question
I may be able to put some effort into this topic in the next 12 months.
The question is
Where is the effort best invested to make screen reader accessible UI?
I see the following options
1. Add screen reader support to ImGui
Pros
ImGui is currently the easiest way to make a UI for vvvv and seems the most widely used. Supporting screen readers here will make it easy for future devvs to have at least default screen reader support in many apps.
Potentially easy for existing apps made with ImGui to be upgraded
Cons
ImGui not intended for making end-user UI. Although it doesn’t seem to stop people. In practice I see lots of published apps using ImGUI so I think it’s still useful to support screen readers here.
I could only add screen reader support for VL wrapping of ImGui, not able to contribute this to the original ImGui project.
ImGui not currently cross-platform.
2. Add screen reader support to Elementa
Pros
It’s the vvvv native UI solution
Cons
Not as developed as ImGui in terms of UI features
Elementa not currently cross-platform
Open Questions
I get the feeling Elementa is not widely used at the moment. For me I can say I haven’t used it in a while. I had some performance issues with Elementa and then ImGui came along and the performance and overall paradigm worked better for me. I haven’t had a usecase to return to Elementa yet.
Are others working on/planning projects with Elementa?
3. Wrap for VL an existing cross platform UI framework that has accessibility features already like MAUI/Xamarin
Pros
Cross Platform!
Cons
Don’t even know where to start with this
Open Questions
Why hasn’t MAUI/Xamarin already been seen in VL? (Has it?) I’ve never tried, but I assume there’s some difficulty here given that MAUI/Xamarin is a default option Microsoft offers for UI but not a default option offered in the vvvv world. Can anyone provide more info?
4. Provide ‘low level’ screen reader support with examples and leave it up to coders to implement custom in every project
Pros
Less effort to develop the framework
Probably have to do this anyway for the above solutions, may as well provide the low level implementation for coders who make completely custom UIs.
Cons
Without integration into a standard UI framework then only apps that take special effort to support screen readers will have any support at all.
5. Embedded Browser-based UI solution
Pros
Already lots of accessibility solutions and information for web UI
Could integrate rabbit-control which also easily enables UI to be run in a remote brower.
Cons
CEF being cross platform in future relies on underlying binding CEFglue being crossplatform, currently supports MacOS but not Linux.
I see less possibilities to really make a node based solution for patching a UI like this, you probably have to get your hands dirty with html. (Although nodes to HTML is something that you could use Cables.gl for)
Open Questions
Any idea how to get the CEF content in stride or skia windows discoverable by screen readers?
Is it possible to run a transparent chromium UI on top of a Skia or Stride app with decent performance?
I think there are already some difficulties communicating between embedded chromium dynamic javascript things and the host app. Anyone know if these issues are showstoppers?
Generally open question @Devvvvs is there a plan for a cross-platform UI framework you wish to support in future e.g. for export to windows and linux?
@Everyone I’m also completely open to other ideas, have you heard of other frameworks etc?
being completely ignorant of the topic… can you provide some pointers as to what it means to support this? are there generic (or probably os specific?) libraries/apis available to implement?
some thoughts:
“ImGui not intended for making end-user UI” - i think the way to read this is: on end-user-facing UIs you’ll likely run into restrictions using ImGui and you should then not complain about that fact. but if you just don’t see any restrictions, it is a perfectly good choice.
ImGui/Elementa not currently cross-platform. well,… they actually are, it is just that vvvv doesn’t provide an out-of the box way to open a corresponding rendering window on other platforms at this point
without knowing anything, option 4. is the way i’d intuitively favor. it seems to be the quickest way to get something working… and then take it from there.
re options 5: you’re saying you have a website that works with a screen reader, but when viewing it through VL.CEF it doesn’t work?
Afaik assistive technologies like screen readers rely on the DOM but VL.CEF renders offscreen on a canvas which is just a bunch of pixels.
I think this might be a problem with all mentioned technologies in the context of vvvv.
accessibility / integration with assistive technologies has luckily become a major topic in ui/ux in the last decade so much that it will be enforced by an EU directive beginning next summer. meaning, if one creates something either directly for institutions of the EU, something for a officials or governmental institutions or want to get funding from the EU, the product has to fullfill a couple of accessibility standards.
all major OSes have built in tools already (windows: narrator, android: TalkBack, apple VoiceOver) so most lectures on (web)ui design/development include a session to try those oneselves. e.g. turn off monitor, try search for something on wikipedia or even try to order something on amazon with keyboard and voiceover only.
as summarized by the first comment in the following imgui issue basic things required are
navigation with keyboard especially Tab
api for assistive technology to access the gui as tree
Press tab a few times and notice how you move between interactive elements and NVDA reads them out
Now open up equivalent vvvv windows and test them.
Complete accessibilty testing is a lot more involved but that should give a basic idea if a common screen reader can at least discover the UI elements.
Compare the accessibility of apps that use windows native controls like 7zip or filezilla to a vvvv window (even the gamma launcher)
Libraries and APIs
In terms of how accessibility technology discovers UI on a technical level I found this series of deep dive articles by Neill Hadder. They are also quite funny
Interesting quote from these articles
As in 1995, accessibility continues to rely for the most part on developers’ use of standard operating system components.
Through those articles discovered what seems to be an attempt at cross-platform access API
Wishlist
From the discussion so far here’s my feeling on what to try
Look into a possibility of a quick win with VL.CEF. Maybe as simple as some init setting to enable accessibility? Looking for github issues etc seems a bit inconclusive if CEF has this possibility or not.
Check out MAUI and effort needed to really wrap it for VL as nicely as say ImGui is now. If that effort could get us a whole new UI library and that library has accessibility by default, then it seems like a good use of effort.
If it’s too complex then look into Tolk (thanks @bjoern) or AccessKit as a ‘low level’ solution that patchers can add to their apps. Then demo how to add accessibility this way to an ImGui app.
Hopefully I can use some time to work this after August. If anyone else wants to get on board more than welcome.
We are using VL.IO.RCP for a web based UI, which you can use with IBM’s Carbon Design System. It is a very well worked out Design System and is very accessible.
What I am more curious about though is what kind of UI’s you are thinking about. Most of the UIs I see made with ImGui are controlling parameters or sometimes for things like settings.
As a UX Designer myself I would be curious what you are intending to use this for and I think you’d have to do some research with some real people using Screen Readers to find out what their needs are around what you are building. I am all for accessibility, but just making sure a UI is screen-reader compatible is only half the story. What are you trying to solve that people using a screen reader are struggling with?
Sure, if you want to use the actual features of Carbon Design System like layout, colors, etc. than it’s probably lots of java scripting.
If you want to just replace the normal RCP TS Client UI elements with the ones from Carbon than it is pretty straight forward. But then you will have pretty automatic layout in 1 column, which is basically just good for mobile.
