Coronavirus-related anxiety, stress can be relieved by AI-based music recommendation technology
Author: Amit Sternberg, Co-Founder and CEO There are many debates over the methods and mathematical models by which doctors and epidemiologists measure, quantify and predict the spreading rate of the Covid-19. However, there seems to be no argument about one fact: many people worldwide suffer from a certain level of stress and anxiety induced by the pandemic.
There is also no disagreement that Covid-19 has dramatically increased the world's attention to the need to maintain a strong and resilient immune system.[1] The strength of which is proven to be the main protective shield we have from Covid-19, or from any future virus that humanity will have to deal with.[2]
While the world is currently focused on trying to find a vaccine against and treatments for Covid-19, it is becoming clear that this viral strain will be followed by a wave of mental problems, which are already being observed, and which we can safely presume will linger for an extended period of time after a vaccine, and/or treatment, are found.
Rubato believes that this crisis will change the world’s consciousness and allow new ways of thinking and new behavioral patterns. We believe it is possible and essential to turn this moment into a new revolution — the Well-being Revolution, where personal health trends are promoted, and there is more awareness of daily stress and anxiety, and, along with this awareness, there is a desire to cultivate and improve the immune system and will accelerate the urge to find a mass-scaled solution for these challenges.[3]
Matching music to the listeners’ physiological and psychological state
Rubato’s innovative technology matches music to the listener’s physiological and psychological state by utilizing algorithms and machine-learned data, based on biometrics and musical attributes. The startup’s mission is to recommend music that scientifically optimizes wellbeing, while also exposing individuals to a new repertoire of music.
Music has prominent effects on humans, but it affects each one of us in different ways: the same music might be relaxing for one person but could increase stress in another; it might increase one’s aerobic performance but contribute to another’s weakness. Similarly, one piece of music might improve one’s sleep quality, but might also prevent others from falling asleep.
Rubato believes that your playlist should be curated based on your personal physiological and psychological needs and responses, rather than by e-commerce algorithms that focus on selling you content.
Our curation, recommendation, and selection of music is based on scientifically-verified measures evidenced in hundreds of scientific studies and algorithms that were developed and tested internally, based on preliminary accumulated data. We facilitate the BPM Entrainment effect through heart, and heart rate, trends. We use Heart Rate Variability (HRV) indexes to determine such preferences as types of scales (major/minor), the emotional influence of text and lyrics, and response to various musical structural patterns.
We use DNN (Deep Neural Networks) to identify and cluster the different musical attributes, as well as biometric vectors in large scale, to quantify the music’s effect on stress, anxiety, sleep, and fitness.
Our process analyzes each musical track for several distinct musical features, such as danceability, acousticness, liveness and more. We note the structure of the song and pay special attention to the mode (major vs minor), the tempo, and even the time signature. By collecting these features together, we are able to build a deep representation of each song’s unique fingerprint, but also to find songs that share similar traits.
Wearable tech to monitor stress
We rely on wearable technology to monitor the biological markers most tied to stress management - elevated stress level measured through the Heart Rate Variability, elevated heart rate and trend changes in heart rate.
Synthesizing these separate data sources allows us to make the necessary connection between a musical track and its effect on the listener’s stress level. As a user listens to songs, our proprietary AI tech incrementally learns how the listener’s body responds to specific types of music, building a comprehensive, individualized map of which musical attributes are most beneficial for a specific individual in order to reduce stress. Based on this understanding, our algorithm recommends new music to help alleviate stress and contribute to achieving a calmer state of mind for the listener.
We leverage these measurements to further fine-tune our proprietary deep neural network model, which consists of many LSTM (long short-term memory) layers that jointly learn from timeseries representing the song data and single-mode timeseries Heart Rate (HR) and HRV data, combining these inputs with musical features extracted from a musical piece to predict a series of physiological indicators that serve as a proxy for psychological states (e.g. “calmness”, “relaxation”, “concentration”). Our final model assigns a stress-reduction score to any song for any user in our system.
We assess the model’s predictive accuracy by measuring HR and HRV parameters collected by our users’ wearables, while they are listening to music pieces explicitly suggested by the previously trained model. Our preliminary test results point to significant correlation between the musical features of a song and the favorable bio-psychological effects associated with stress reduction and show promising predictive accuracy for our recommendation engine.
For many years, people have been listening to music for their enjoyment only. Now, when the increasing trend of biohacking enables each one of us to become a better version of himself, people will be able to use music as a form of Biofeedback, and curate playlists that would scientifically help them optimize their health and overall wellbeing.
***
About Rubato:
Rubato is an innovative technology that matches music to the listener’s physiological and psychological state. We are utilizing algorithms and machine-learned data, based on biometrics and musical attributes. Our mission is to recommend music that scientifically helps people to manage their stress and improve sleep quality, by turning enjoyable music listening experience into scientific Biofeedback treatment.
***
[1] https://www.post-gazette.com/ae/music/2020/04/20/immune-system-music-therapy-health-benefit-study-mind-body-COVID-19/stories/202004150125 [2] https://www.sciencedirect.com/science/article/pii/S1876034120305670 [3] https://www.sciencedirect.com/science/article/abs/pii/S1471489216300558