Can Article Success Be Predicted? I Started With 399 Finance Articles

I Rated 399 Medium Finance Articles. Could AI Tell Which Ones Performed Better?

I compared my own judgments of article previews with AI scores and public engagement. There was some signal, but less than a writer might hope.

Before publishing an article on Medium, you have already made several predictions.

You decided that an idea was worth writing. You chose a title because you thought it would attract attention. You wrote a subtitle and selected a thumbnail because you expected them to give someone a reason to click.

Usually, those decisions come down to intuition.

This title feels stronger.

That preview looks interesting.

I would probably read this one.

I wanted to know whether those judgments could be measured, and whether they had any relationship with how articles actually performed.

So I worked with 399 Medium finance articles, rated their previews myself, and had AI evaluate them independently. Then I compared those ratings with the articles’ publicly visible engagement.

The question was straightforward:

Do human and AI judgments of an article’s preview contain useful information about its performance?

The answer was a qualified yes. Some ratings were associated with stronger engagement, but the relationships were modest. And the score that sounded most like an overall verdict was not necessarily the most useful one.

What I collected

The 399 articles came from a larger dataset of 1,794 canonical Medium articles collected through observations of tag pages.

The smaller group was the subset for which I had a validated thumbnail and a complete set of human preview ratings. This gave me consistent material to compare: article titles, subtitles where available, and the actual images used during the rating exercise.

For each article, I recorded five human judgments.

Personal click appeal measured how much I wanted to open the article.

Title hook strength measured how compelling the title felt on its own.

Visual hook measured how effectively the thumbnail caught my attention.

Emotional pull measured how much curiosity, concern, aspiration or other emotion the preview created.

AI or low effort appearance recorded whether the preview appeared generic, AI generated or low effort.

The title hook pass showed only the title. The other dimensions used the relevant preview components.

The ratings were blind: the interface hid claps, responses, publication information, article age and AI scores.

That mattered. Once you know that an article has performed well, it becomes much easier to find reasons why its title was supposedly brilliant.

I wanted to record my judgment before seeing that information.

There is an obvious limitation here: the human rater was me. These ratings describe one person’s reactions, not the preferences of the average Medium reader.

What the AI evaluated

I also used GPT 5 mini to score the same 399 articles under three text conditions: the title alone, the subtitle alone, and the title and subtitle together.

Each condition had 399 completed scores, giving me 1,197 text scoring records.

The AI assessed dimensions such as emotional pull, expected clap potential, expected comment potential and overall article potential.

The scoring workflow excluded performance information from the inputs. The model judged the text without receiving the article’s claps, responses, age or engagement label.

There were also three completed image scoring passes over the same 399 articles: thumbnail alone, title plus thumbnail, and title plus subtitle plus thumbnail.

Those image scores survive in the dataset, but the verified results I report here focus on the text scores and my human ratings. Having collected more scores does not, by itself, establish that they improve prediction.

What “performance” means in this experiment

For someone else’s Medium article, I cannot see all the metrics I would ideally want.

Public engagement does not tell me how many people saw a preview, how many clicked, how many finished reading or how much money the article earned.

For this analysis, I used the information available: claps and responses.

The engagement score was:

Engagement score = ln(1 + claps) + 2 × ln(1 + responses)

The logarithms reduce the influence of very large counts. The formula also gives responses twice the weight of claps on the logarithmic scale.

That weighting is an analytical choice. It reflects an interest in discussion as well as applause; it is not an established definition of success on Medium.

Throughout this article, “performance” therefore means this particular public engagement measure. It does not mean readership, reader satisfaction or earnings.

Age is another complication. Older articles have generally had more time to accumulate engagement.

Of the 399 rated articles, 332 had a usable inferred publication date, while 67 did not. That gives me a possible starting point for an analysis that accounts for age, but the statistics below use engagement without age adjustment across all 399 articles.

They should be read with that limitation in mind.

Which individual ratings tracked engagement?

I first looked at whether higher ratings tended to go with higher engagement.

For this, I used Spearman correlation, which compares rankings. A positive value means articles with higher ratings tended to rank higher on engagement too.

A value near zero indicates little relationship between the rankings. A value near one would indicate a very strong relationship.

Several relevant results were:

AI emotional pull from the title alone: 0.2443.

AI comment potential from the title and subtitle: 0.2418.

AI emotional pull from the title and subtitle: 0.2393.

My human rating of preview emotional pull: 0.2265.

AI overall article potential from the title and subtitle: 0.1820.

All five comparisons used 399 articles.

These are modest positive associations.

They suggest that the ratings captured something related to engagement, but they leave plenty of room for articles that looked promising and performed poorly, and articles that looked ordinary and performed well.

One result I found interesting was the consistency around emotional pull.

The AI rating of the title alone, the AI rating of the title and subtitle, and my own preview rating all showed correlations in roughly the same range.

That is compatible with the idea that emotionally engaging previews contain useful information. It does not show that adding emotional language would cause an article to perform better.

A preview might feel emotionally compelling because it addresses a topic people already care about. The article might also come from an author with a large audience or receive strong distribution. This comparison cannot separate those explanations.

Could the AI scores distinguish higher performing articles?

Correlation is one way to examine the ratings. Another is to ask whether a score tended to rank articles with stronger engagement above the rest.

For this comparison, the higher performing group consisted of articles that qualified for the top 20% engagement label defined across the full collection of 1,794 articles.

The 399 article subset contained 175 articles with that label, or approximately 43.9%.

That proportion is important. The rated subset was not a representative cross section of the larger collection: it contained a substantially higher share of articles carrying the high engagement label.

I evaluated the individual AI scores using AUC, a measure of ranking discrimination.

An AUC of 0.50 corresponds to random ranking. An AUC of 1.00 would mean perfect separation of the two groups.

The verified results were:

Emotional pull from the title alone: AUC 0.5945.

Comment potential from the title and subtitle: AUC 0.5932.

Emotional pull from the title and subtitle: AUC 0.5773.

Overall article potential from the title and subtitle: AUC 0.5553.

Overall article potential from the title alone: AUC 0.5086.

These are evaluations of individual scores, not cross validated results from a trained model combining multiple features.

For an intuitive interpretation, an AUC of 0.5945 means the emotional pull score from the title alone would rank a randomly selected article in the higher performing group above a randomly selected article outside that group about 59.5% of the time, counting tied scores as half a win.

That is some discrimination, but it is a long way from a dependable success detector.

“Overall potential” was a weak verdict

The result that stood out most to me was the gap between a specific dimension and a broad judgment.

When the AI received only the title, emotional pull reached an AUC of 0.5945, while overall article potential reached an AUC of 0.5086.

The overall verdict was very close to random ranking in this sample.

Adding the subtitle improved the overall potential score to 0.5553, but it still remained below the reported emotional pull and comment potential scores.

This makes me cautious about asking an AI a question such as:

“How successful will this article be?”

An answer can sound decisive while containing little useful ranking information.

A narrower question, such as how much emotional pull a title creates, may be easier to assess consistently. In this experiment, that narrower score was more informative.

It still was not accurate enough to make an editorial decision automatically.

Subtitles did not improve every score

I expected that giving the AI more information might help.

Sometimes it did. The overall potential score discriminated better with a title and subtitle than with a title alone.

But emotional pull AUC moved in the other direction: 0.5945 for the title alone, compared with 0.5773 for the title and subtitle.

It does show why “more context must be better” is an assumption worth testing.

Different inputs can change how a model judges an article. Whether that change helps depends on the particular score and outcome.

My own intuition also had limits

My human preview ratings showed modest relationships with engagement.

Emotional pull had a Spearman correlation of 0.2265.

Personal click appeal had a correlation of 0.1363.

Title hook strength had a correlation of 0.1230.

Visual hook had a correlation of 0.1179.

My emotional reaction to a preview tracked engagement more closely than my reported desire to click it.

That is a useful distinction for me as a writer.

What I personally want to read may differ from what attracts engagement within the audience and distribution conditions represented in this collection.

What this changes about my writing process

I would use these ratings to structure a decision, rather than to outsource it.

If I am comparing several article concepts, I can record what I think about them before publication. I can ask AI to assess specific dimensions and examine where its judgments differ from mine.

A disagreement gives me something concrete to consider.

Perhaps the title has emotional pull but makes an unclear promise. Perhaps I like the subject personally, but the preview gives another reader little reason to care.

The useful part is making those assumptions visible.

What I still need to test

The main limitations are straightforward.

This was a retrospective collection of Medium finance articles, with one human rater and a selected subset of 399 previews. The outcome used public engagement, and the reported statistics were not adjusted for article age.

Author audience, publication distribution, topic popularity and article quality could all contribute to the observed relationships.

I also evaluated multiple dimensions. These are exploratory findings, not a final benchmark.

The next useful step would be to define a prediction method in advance, record forecasts for new articles, and evaluate them over a consistent period after publication.

A separate experiment could test alternative titles or thumbnails under comparable conditions. That would help answer whether changing a preview changes performance.

For now, I have a more modest result:

Some human and AI preview ratings contain information about public engagement, but neither provides a strong standalone verdict on which Medium finance articles will succeed.

The most useful lesson may be that a specific question can produce a better signal than a grand prediction.

“Does this title create emotional pull?” gave me more information than “Is this article likely to succeed?”

That is enough to make the scoring process worth investigating further. It is also enough to remind me how much those scores still miss.