How to Use Data Analytics for Effective Policy-making

So here is how the story began :

Millions of research papers are published each year. However, due to the pandemic, most journals were flooded with COVID-based publications.

Journals saw this as an opportunity to cash on while Ph.D. students sighed a relief that they would soon get a degree due to a higher publication percentage of COVID-based research papers.

According to the dimensions database prediction for 2020, scientists could publish over 2 million COVID-based papers by Dec 2020.

To be honest, this figure of approximately 2 million covid-19 research papers around 2020 is true. I collected a corpus of 1 million coronavirus literature to find the domain areas most impacted by the pandemic.

The idea was simple but interesting:

Since journals are overloaded with covid-19 research papers, why not do data analysis to look for trends in different domains and how the pandemic impacted these fields.

NOTE: Since the Publication is under review I can't go into detail about the exact methodology I used to find trends

However, what I found most interesting was the use of a Natural Language Processing technique called Latent Dirichlet Allocation (LDA) Topic Modeling.

People often confuse LDA of natural language processing with LDA of machine learning. However, both are different. 

Linear Discriminant Analysis (LDA) in Machine Learning is a supervised classification technique whereas Latent Dirichlet Allocation (LDA) in Natural Language Processing is an unsupervised classification technique.

Data Analytics helps enhance Decision Making

But How?

I often ask people what is the most important thing in data analysis. 80% of the time, they say of course it's Data. 

Guess what, Data comes second. First comes your QUESTION. 

What do you want to analyze? What is the Problem being faced?

Some example Questions are :

If employees receive frequent breaks will they work for more hours?

If I study hard will I get better grades?

Once you have the RIGHT QUESTION, you are good to go. 

RIGHT QUESTION means you have enough data on the problem to be analyzed. 

You better change the Question if data is not easily available.

MY RIGHT QUESTION:

Before collecting covid-19 research papers, I asked this Question:

What are the domain areas most impacted by the pandemic?

Once I knew the Right Question, I instantly have an approximate idea of what kind of data to collect to answer the question.

HOW DATA ANALYTICS ACED IT

After performing all the necessary steps of text preprocessing, and application of machine learning models, a deep learning technique, and Natural Language Processing techniques, I found the following answer:

The beauty of LDA is that it gives you topics as domains that are further preprocessed and refined to make the best possible decisions.

Furthermore, analyzing the monthly and weekly domain trends will help policymakers, businesses, and concerned institutions/organizations to direct attention toward the most crucial areas.

From the above bar graph, it is evident that besides mortality, anxiety was also an issue that people faced during the pandemic. The solution is that the governments should give attention to the mental health sector by giving incentives to the healthcare workers and increasing funding for the healthcare sector.

The same methodology can be used to find the domain areas most impacted in the case of the current scenario where we see quickly rising cases of monkeypox virus. Once the dataset is collected we can analyze the domains most impacted by yet another looming deadly virus.

Conclusion

If you can ask the right question, you will collect the right data.

If you have the right data, you can make the best decisions for your future using data analytics.

And better decisions will help us all in the long term.

I hope you learned something

Keep learning

Keep Growing

See ya soon!