On January 27, 2020, Planned Parenthood released an update (version 3.1.8) to Spot On, which is a mobile app that allows users to track their period and birth control method. Unfortunately, many users (including myself) were unhappy with the updated version to say the least. In Part 1 of this blog series, I described my own experience with the updated version.
Here in Part 2 I evaluate and compare the updated version of Spot On and the previous version with the Mobile Application Rating Scale (MARS). MARS is a tool that appraises an app’s quality within four different dimensions; engagement, functionality, aesthetics, and information. MARS also includes an app subjective rating. All MARS items are rated on a 5-point Likert scale ranging from 1 (inadequate) to 5 (excellent).
I fully anticipate that Planned Parenthood will be making changes to Spot On to address the feedback they’ve been receiving. As such, it is important to note that this evaluation pertains to the app as it functioned/appeared on the day I conducted its MARS evaluation (below). Furthermore, this is just my evaluation of Spot On. Some users may have better or worse experiences than me with this app.
|“New Version”||“Old Version”|
|Date MARS Completed:||January 31, 2020||February 1, 2020|
|Version:||3.1.8 (released Jan. 27, 2020)||APK version 2.9.1 (update April 6, 2019)|
App Quality Rating for Engagement:
Overall both versions of Spot On did well for engagement, with the new version scoring an average of 4/5 and the older version scoring 4.5/5. While the app contains many options for customization, the big difference for me between the two versions was that in the update the app did not retain a lot of these customization. Much of this is likely due to bugs. Also, the social media feel of the newer version did not appeal to me personally.
Entertainment: Is the app fun/entertaining to use? Does it use any strategies to increase engagement through entertainment?
|New||5||Highly entertaining and fun, would stimulate repeat use.|
|Old||5||Highly entertaining and fun, would stimulate repeat use.|
Comments: While the app may not be about “entertainment” per se, it is definitely the type of app that someone would use repeatedly. The use of emojis is cute and makes the app feel more personal and fun.
Interest: Is the app interesting to use? Does is use any strategies to increase engagement by presenting its content in an interesting way?
|New||5||Very interesting, would engage user in repeat use.|
|Old||5||Very interesting, would engage user in repeat use.|
Comments: Both versions of Spot On do a great job of presenting the content in an interesting way, and I believe in this regard would encourage repeated use.
Customization: Does it provide/retain all necessary settings/preferences for app features (e.g., sound, content, notifications, etc.)?
|New||3||Allows basic customization to function adequately.|
|Old||5||Allows complete tailoring to the individual’s characteristics/preferences, retains all settings.|
Comments: Both versions of Spot On allow for quite a bit of customization, such as type of birth control being tracked (including the pill, patch, ring, shot, IUD, and implant), and custom moods, activities, and symptoms. Users can also choose whether they want to receive birth control and appointment reminders. It is really great to see this level of customization.
However, one of the areas the new version of the app doesn’t do as well on is in retaining some of these customizations, which is considered in the MARS evaluation. Namely many of these customizations were not retained when data was transferred to the new version. For example, information about my birth control didn’t transfer over properly (and the app was telling me to start a new pack every couple of weeks), and I couldn’t access details about my menstrual flow for previous cycles even though I had logged this information. Some of the reviews I read reported that some users weren’t able to transfer their data and lost their customized content all together. While I suspect much of this will be improved as the developers fix some of the bugs (just like they fixed the broken birth control reminders for this version), as the app currently stands this is a pretty big issue. For that I give it a 3 with the hopes with will be fixed and become a 5 for me like the older version.
Interactivity: Does it allow user input, provide feedback, contain prompts (reminders, sharing options, notifications, etc)? Note: these functions need to be customizable and not overwhelming in order to be perfect.
|New||4||Offers a variety of interactive features/feedback/user input options.|
|Old||4||Offers a variety of interactive features/feedback/user input options.|
Comments: Both versions of Spot On are quite interactive. However, there are difficulties in accessing some of the information I’ve input. Since you can’t edit logs if you go too far into the past, you cannot quickly figure out which each emoji stands for if you forget. In such a situation you’d have to scroll forward to a more recent day so you can open up the log and figure out what the emoji represents. And for menstrual flow this information is completely lost.
However, the older version had some issues too. While these features were overall better, my birth control pill reminder was inconsistent and only seemed to work if I was connected to data. And even then I sometimes wouldn’t get a reminder. This is a pretty big flaw for an app that is supposed to give your prompts to take your birth control, so I can’t really give it a 5 here either.
Target group: Is the app content (visual information, language, design) appropriate for your target audience?
|New||3||Acceptable but not targeted. May be inappropriate/unclear/confusing.|
|Old||4||Well targeted, with negligible issues.|
Comments: The app has (in my opinion) done a good job here in some regards. It appears to be very gender neutral and the language seems appropriate. However, the new design is confusing and feels like it is targeting a much younger audience than me. (Yeah, I know, okay boomer). The app overall has more of a social media look, which – to me at least – seems a bit odd given the app’s stated purpose. Perhaps their target audience is now teenagers, but that excludes a large part of the population who may want to use the app to track their menstruation, fertility, and contraceptive method. The previous version felt a bit more inclusive of a broader audience, although it also used a lot of emojis which potentially some older users may be less familiar with. However, this version felt fairly well targeted in comparison.
App Quality Rating for Functionality:
Overall I gave the new version a 4/5 compared to a 4.75/5 for the older version for functionality. I anticipate the new version’s performance will improve with time, as there were a lot of bugs when I tried to use it. However, a big difference between the two versions for me was that I found the new version less intuitive and more complicated for tracking. While the previous version allowed me to do all of my tracking on the home screen, there are more steps to tracking in the updated version.
Performance: How accurately/fast do the app features (functions) and components (buttons/menus) work?
|New||3||App works overall. Some technical problems need fixing/slow at times.|
|Old||4||Mostly functional with minor/negligible problems.|
Comments: I found the new version of Spot On to be a bit slow to load, although this isn’t a big deterrent for me personally. However, there were a number of accuracy issues; for example, the home screen not recognizing I had started a new pill pack no matter how many times I indicated that I had. It also gave me some generally incorrect information about my birth control and period, although I think a lot of this was due to bugs when my data was transferred over to the new version. The app works quite quickly once it’s loaded. However, it crashed on me several times, which is an issue I never had with the older version.
I only gave the older version a 4 because the birth control reminder worked inconsistently for me, so I can’t say it was perfect.
Ease of use: How easy is it to learn how to use the app: How clear are the menu labels/icons and instructions?
|New||3||Useable after some time/effort.|
|Old||5||Able to use app immediately; intuitive, simple.|
Comments: I found it took me some time to figure how to use the new version. That being said, I think for me this was largely because I was so used to the old version. Planned Parenthood has released information explaining the new features and how to use the app, which is great. I also really liked that the calendar is located in the menu at the bottom of the screen, as I sometimes would forget where to access it on the older version. However, I found features like tracking activities, moods, and symptoms is not as simplistic and clear as they used to be. I personally found the older version more intuitive, but different users may have a different experience.
Navigation: Is moving between screens logical/accurate/appropriate/uninterrupted; are all necessary screen links present?
|New||5||Perfectly logical, easy, clear and intuitive screen flow throughout, or offers shortcuts.|
|Old||5||Perfectly logical, easy, clear and intuitive screen flow throughout, or offers shortcuts.|
Comments: I didn’t find navigation to be an issue on either version of Spot On.
Gestural design: Are interactions (taps/swipes/pinches/scrolls) consistent and intuitive across all components/screens?
|New||5||Perfectly consistent and intuitive.|
|Old||5||Perfectly consistent and intuitive.|
Comments: I really didn’t find any issues here for either version. This is very consistent throughout both versions of the app.
App Quality Rating for Aesthetics:
For aesthetics I gave the new version an overall score of 4.3/5, and the older version 4.7/5. I have to say that overall the newer version is much flashier and looks really sleek. However, it lost points due to the layout and it being difficult to find somethings. Otherwise the new version would have scored higher than the older version.
Layout: Is arrangement and size of buttons/icons/menus/content on the screen appropriate or zoomable if needed?
|New||3||Satisfactory, few problems with selecting/locating/seeing/reading items or with minor screen size problems.|
|Old||5||Professional, simple, clear, orderly, logically organized, device display optimized. Every design component has a purpose.|
Comments: I personally don’t like the layout for the ‘trackables’ like mood, activity, etc. Previously this could all be done on the home screen, and each of these categories had its own little bubble at the bottom of the screen. Tapping on it would bring up response options only for that category, and the emojis you selected would then appear in that bubble on the home screen. Now you have to open the Calendar, tap the day you want to log these for, and hit the “+” to bring up everything for all categories. Not as convenient in my opinion. There is a “Quick Log” on the home screen of the new version, but this just gives users three options (that the app picks). Neither version has a zoom function, which could be problematic for those with a visual disability. I think this is more of an issue with the new version because a lot of the text seemed smaller. Some parts the text get cut off in the new version (e.g., the “birth control no longer working” screen). Despite these critiques, there are also some very positive design elements with regards to the layout; for example, the app doesn’t appear cluttered at all.
While the older version certainly isn’t as pretty as the new version, I (personally) prefer this layout. It feels very logical and simple to me, and displays properly on my device. I also find the text easier to read.
Graphics: How high is the quality/resolution of graphics used for buttons/icons/menus/content?
|New||5||Very high quality/resolution graphics and visual design – proportionate, stylistically consistent throughout.|
|Old||5||Very high quality/resolution graphics and visual design – proportionate, stylistically consistent throughout.|
Comments: The graphics are excellent. I think they’ve done a particularly nice job on the new version. The new version is definitely flashier, but the graphics were still great on the old version.
Visual Appeal: How good does the app look?
|New||5||Like 4 but also very attractive, memorable, stands out; use of colour enhances app features/menus.|
|Old||4||High level of visual appeal – seamless graphics – consistent and professionally designed.|
Comments: As stated above, the new version is much flashier than the old version. I’m not a fan of the social media feel that the app has, but overall it is visually appealing.
App Quality Rating for Information:
I gave the new version an overall score of 3.75/5 for information compared to 4.25/5 for the older version. While I found the information contained in the additional resources to be better in the newer app, a lot of the information given that was specific to me was wildly incorrect. I suspect most of this is due to bugs, and if fixed I would score the updated version higher than the older version with regards to information.
Accuracy of the app description (in app store): Does app contain what it described?
|New||2||Inaccurate. App contains very few of the described components/functions.|
|Old||4||Accurate. App contains most of the described components/functions.|
Comments: While the updated version contains the features described in the app store, in my experience there was a lot of functionality issues for some of the features and they did not work for me. For example, the app store describes that the app can accurately predict my period and tracking my birth control, but in the updated version this is no longer the case (once again, largely due to bugs I suspect). Furthermore, while I can input details about my menstrual flow – as described in the app description – I cannot access this information if I go too far into the past despite having entered these details. So while the app technically contains these features, they currently aren’t accurate and lack some of the functionality one would expect from the description. Some may think a 2 seems unfair, but these are pretty fundamental features for an app of this nature. If this gets fixed, I would change my rating to a 4 or 5.
I gave the older version a 4 because the birth control reminder has never worked consistently for me, and a user would expect this to work properly based on the app description. But otherwise it’s pretty accurate.
Goals: Does the app have specific, measurable, achievable goals (specified in the app store description or within the app itself?). N/A
Comments: While I would say the app has an overall goal of promoting and supporting sexual health, this item felt not applicable since it doesn’t have specific, measurable goals.
Quality of information: Is app content correct, well written, and relevant to the goal/topic of the app?
|Old||5||Highly relevant, appropriate, coherent, and correct.|
Comments: This is a tough one to rate. Both versions have a lot of additional information and resources for users. For example, additional information on birth control, fertility, periods, sex, etc. The new version even has a section on consent, which is amazing to see! Both versions have a glossary of words related to sexual/reproductive health. From the sections I’ve read through this content all sounds pretty good. However, as previously described some of the information specific to me was incorrect (e.g., when to start a new pill pack, what to do if I missed my pill, and general information on my menstrual cycle). I suspect this will all be fixed in time, but these are pretty big mistakes. I had to take the new version down from a 5 to a 4 for that.
Quantity of information: Is the extent coverage within the scope of the app: and comprehensive but concise?
|New||5||Comprehensive and concise; contains links to more information and resources.|
|Old||4||Offers a broad range of information, has some gaps or unnecessary detail; or has no links to more information and resources.|
Comments: There is a lot of information on the app – more so on the updated version – but it is extremely well organized. The new version is a little more concise than the old version and had more links to additional information/resources. I have to say, they have done a fantastic job here.
Visual information: Is visual information of concepts – through charts/graphs/images/videos, etc. – clear, logical, correct? N/A
Comments: I didn’t see any visual information in the resources sections, but it’s entirely possible I missed it.
Credibility: Does the app come from a legitimate source (specified in app store description or within the app itself?)
|New||4||Developed by government, university or [small NGO/institution] but larger in scale.|
|Old||4||Developed by government, university or [small NGO/institution] but larger in scale.|
Comments: I’m not sure if they applied for some sort of research funding for this, so I put it at a 4. While I know some people have an issue with Planned Parenthood, I would say they’re a very credible source on this topic.
Evidence base: Has the app been trialled/tested: must be verified by evidence (in published scientific literature)? N/A
Comments: To my knowledge this app (particularly the update) hasn’t been tested scientifically speaking.
App Subjective Quality:
In the previous sections pertaining to the app quality, the newer version actually didn’t do too poorly. In fact, it scored much higher than I thought it would. A lot of the issues that the previous section of MARS picked up on were bugs, which in time the development team will be able to iron out. It’s unfortunate that these weren’t caught before the launch, and as a researcher I think more time should have gone into testing the app.
This section – the app subjective quality – is where the new version of the app really doesn’t do so well for me. Some of this is due to my own personal preferences, and other elements just didn’t really seem to get picked up by MARS as I applied it to both versions of Spot On. Here the newer version scored substantially lower than the older version, receiving a 1.25/5 compared to a 4.5/5.
Recommendation: Would you recommend this app to people who might benefit from it?
|New||2||There are very few people I would recommend this app to.|
|Old||5||I would recommend this app to everyone.|
Comments: While I generally wouldn’t recommend this app as it currently stands, I may still recommend it to some people depending on their health goals and purposes for using the app. The older version I used to recommend to all of my menstruating friends.
Frequency of Use: How many times do you think you would use this app in the next 12 months if it was relevant to you?
Comments: While the app is certainly relevant to me, I wouldn’t use the new version as it currently stands. In fact, I uninstalled the new version after I finished writing its MARS evaluation and found a way to reinstall version 2.9.1. The old version gets a 5 here because I use it every day.
Purchase: Would you pay for this app?
Comments: If they combined the some of the new improvements (e.g., improved aesthetics, expanded resources) with the features I liked in the old version, I would absolutely pay for this app. Oh, and they would have to remove the social media vibe that the app now gives off. But we’re going to get into that in the next section.
Star Rating: What is your overall star rating of the app?
|New||1||One of the worst apps I’ve used.|
|Old||5||One of the best apps I’ve used.|
Comments: While the old version wasn’t perfect (e.g., inconsistent BCP reminders), it worked really well for my goals and purposes for using the app. I have tried a few other period trackers, and this is hands down the best I found for addressing my needs. The new version (as it currently stands) doesn’t meet my needs and has lost the features I liked.
As I talked about in Part 1, the new version has more of a social media vibe, which is a big reason why I would only give it 1 star. I feel this is a disconnect with the app’s purpose; tracking menstruation and birth control methods. The home screen in particular feels like a newsfeed, broadcasting personal information about my health to anyone sitting near me and then followed by updates from Planned Parenthood as you scroll down. I take major issue with this display of my personal information. As soon as you open the app it displays notification in incredibly large text on brightly coloured backgrounds. This would easily catch the attention – and be legible – to those around me. It’s also impossible to open the app without these messages popping up on the screen since they’re on the home screen. While I am (clearly) pretty comfortable talking about my reproductive health, it is entirely my decision when, how, and to whom I want to share this information. I typically don’t share this information with people on the bus, train, office, etc. Furthermore, other people may not be as comfortable as I am with reproductive health and this could be a deterrent to using a potentially very helpful and educational app.
While the message asking how my period is going is just weird and would probably make me a bit irritable, the loudly broadcasted “birth control is not working” message is quite concerning. I completely get that it’s important for someone to know that their birth control may not be working. But it feels like the app is shaming users with the message, especially considering it’s blasted at you on the home screen for a week. The fact that anyone nearby would be able to read this when the app is opened almost turns it into a public shaming that the individual messed up their contraceptive method.
Spot On MARS Score Card:
|New Version (3.1.8)||Old Version (2.9.1)|
|Engagement Mean Score:||4||4.6|
|Functionality Mean Score:||4||4.75|
|Aesthetics Mean Score:||4.3||4.7|
|Information Mean Score:||3.75||4.25|
|App Quality Mean Score:||4.0||4.6|
|App Subjective Quality Mean Score:||1.25||4.5|