Running a real store as a testbed for commerce software

Commerce software is full of features designed in meeting rooms. They demo well, they fill a pricing page, and they solve problems that store owners rarely have. The problems store owners do have — a variant that sells out at the wrong moment, a refund pattern nobody connected, a bundle that never reaches the warehouse — are messy, specific and easy to miss from the outside.

Oryvelon operates Noveniq, a direct-to-consumer brand selling technology accessories. It is a real store with real products, suppliers, orders and customers. It is also, as a secondary benefit, a practical testbed for what we build at MerchNivo, our AI e-commerce employee for Shopify stores.

This article explains how running a real store as a testbed works in practice, why it is valuable, where its limits are, and — most importantly — how we get the learning without mixing customer data between companies.

Why software teams should run the thing they sell to

There is a long tradition of software built by people who use it. It tends to be sharper, because every rough edge costs the builders time personally. The same logic applies to commerce software, with one twist: to feel what a store owner feels, you need a store.

Interviews with merchants are essential, and we do them. But interviews tell you what people remember and what they think is important. Operating a store tells you what actually happens on a Tuesday morning when a supplier emails to say a shipment is delayed, three customers ask where their orders are and the ad budget is still spending on a product that just went out of stock.

A live store provides three things interviews cannot:

1. Unfiltered problems. You meet the problem before anyone has described it neatly.
2. Real timing. You learn how quickly something needs to be noticed to be useful. A stock alert at 9 a.m. is worth more than the same alert at 6 p.m.
3. Honest usage. When the team running the store ignores an alert, that is a clear verdict on the alert.

Noveniq is a business first

The most important rule is the one people most often break when they set up a "demo store": the testbed must be a real business with its own goals.

A store that exists only to test software drifts. It stops ordering stock when stock is inconvenient. It never has real customer complaints. Its data is too clean and its problems are too convenient. Worse, it tempts the team to judge the software by how well it works on a store designed to make it look good.

Noveniq has its own source-of-truth document, its own revenue targets, its own checkpoints and its own customers. It is evaluated at 30, 60 and 90 days on whether to continue, stop or scale, like every other Oryvelon company. If Noveniq were only useful as a testbed and not viable as a business, that would be a reason to rethink it, not to keep it running as a lab.

This matters for the testbed value too. The problems worth learning from are the ones a real store faces because it is trying to grow and make money. We describe the wider approach to operating a direct brand in a DTC brand as an operating business.

What crosses over, and what does not

This is the part we are most careful about. Oryvelon runs on a principle of shared infrastructure, separate data, and we do not allow cross-product user data. Noveniq's customers bought technology accessories from Noveniq. They did not agree to become part of a software company's dataset.

So the boundary is explicit:

The pattern is simple to state: learning crosses, records do not. A Noveniq operator might tell the MerchNivo team "we missed a refund cluster because the notes were buried; the briefing should summarise refund reasons". That is valuable, and it contains no customer data. What the operator does not do is send the refund notes themselves.

How MerchNivo runs on Noveniq

If Noveniq chooses to use MerchNivo as a customer, it connects in exactly the way any other merchant would: through Shopify's standard access, granted by the store, into its own isolated tenant. MerchNivo processes that data to serve Noveniq, the same way it would serve any store, and Noveniq's data is subject to the same tenant isolation and retention rules as everyone else's.

There is no back door, no special internal access and no bulk export into a shared analytics store. MerchNivo engineers do not get broader access to Noveniq's data because both companies are in the same group. Access follows the same least-privilege rules that apply everywhere.

Test data for development

When MerchNivo needs realistic data to develop and test features, it uses synthetic data shaped like a store's data — plausible products, variants, order patterns and edge cases — not copies of Noveniq's records. Noveniq's experience helps make that synthetic data more realistic ("stores like ours have bundles, multi-location stock and seasonal spikes"), but the data itself is generated. We cover this in environments and test data.

What a testbed is good at

Used well, a real store is extremely good at a specific set of things.

Finding the problems worth solving. Many of the operations signals MerchNivo watches are the kind of thing an operator discovers only by running a store: the order stuck because of one mapping issue, the discount code that kept working after a campaign, the variant that sold out in one colour while the product looked healthy overall.

Testing whether an alert is useful. A signal can be accurate and still useless. A real operator tells you quickly: too late, too noisy, not specific enough, or exactly right.

Setting sensible defaults. What threshold for stock cover feels right? How many refunds before a cluster is worth mentioning? A live store gives a first answer that is grounded in something real, before it is checked more widely.

Catching awkward edge cases. Real stores have messy data: products renamed mid-season, variants archived and restored, locations added and removed. Software that works on clean demo data often breaks on these.

Rehearsing failure. What does the operator see when the AI model is unavailable, or when inventory data looks inconsistent? Watching a real person react to a degraded briefing is more instructive than any test script.

What a testbed cannot prove

It is equally important to be honest about the limits. One store is one data point.

It is not a market. Noveniq sells technology accessories direct to consumers. Its product mix, order volume, seasonality and team size are its own. A fashion brand, a food brand or a high-volume wholesale operation will have different problems. Anything learned from Noveniq is a hypothesis about other stores, not a conclusion.

It is biased towards the builders. The people running Noveniq know the MerchNivo team and understand the product's intent. They are more forgiving and more articulate than a typical merchant. Features that make sense to them may confuse someone who has never heard of the product.

It cannot validate pricing. Whether merchants will pay, and how much, can only be learned from merchants who are paying. A store the group owns tells us nothing about willingness to pay.

It can create false confidence. If every feature is shaped around one store, the product ends up fitting that store and nothing else.

So we treat Noveniq's feedback as a strong early signal and then check it. Before a pattern from Noveniq becomes a default in MerchNivo, we look for the same problem among other merchants through interviews, support conversations and usage patterns within each merchant's own tenant. If nobody else has the problem, it probably does not belong in the product.

How the feedback loop works

In practice, the loop is informal but disciplined. It runs on the same operating cadence as the rest of the group.

1. Noveniq runs its store and notices friction: something missed, something noisy, something slow.
2. The operator writes it down in plain language, without attaching customer records. A short note: what happened, when it would have been useful to know, what they did about it.
3. MerchNivo reviews the notes and decides whether it looks like a general problem, a Noveniq-specific quirk, or a data quality issue.
4. If it looks general, MerchNivo checks with other merchants before building.
5. If it gets built, Noveniq is often among the first to use it, as an ordinary customer in its own tenant.
6. The result is reviewed: did the change help? Did it help anyone else?

A worked example makes the rhythm concrete. Imagine Noveniq runs a weekend promotion on phone cases. On Monday, the operator realises that one popular case sold out on Saturday afternoon while ads kept running until Sunday night. Their note to MerchNivo reads, roughly: "Stock-out on promoted variant was not visible until Monday. During campaigns, stock alerts need to be tighter and faster." There is no customer data in that sentence. MerchNivo asks a few other merchants whether they have had the same experience, finds that several have, and adds a campaign-aware stock threshold. Noveniq then uses the new behaviour during its next promotion, just as any other merchant would.

What the store gets in return

The arrangement is not one-way. A store that sits next to a software team gets something too, and it is worth being clear about what that is, because it should never distort how the store is run.

Noveniq benefits from a team that thinks carefully about operations signals, thresholds and daily routines. The act of writing feedback notes forces the store to reflect on its own processes: why did that bundle fail to map, why did nobody read the refund notes, why was the reorder late. Many of those reflections lead to fixes inside Noveniq that have nothing to do with software at all — a clearer supplier calendar, a better product description, a simple weekly check.

It also gets early access to changes, as any design partner would. But it pays for and uses the software on the same terms as other merchants, and its operators are free to say a feature is not worth using. A testbed that is obliged to praise the product is worse than no testbed.

Failure modes of internal testbeds

We have thought hard about how arrangements like this go wrong, because the failure modes are predictable.

The store becomes a showroom. Its data gets cleaned up so the software looks good in demos. Stock counts are corrected by hand before a presentation. Real problems are hidden. The fix is to never use the testbed as a sales demo; demos use synthetic data.

Data quietly leaks across. Someone exports "just a sample" of orders to debug a feature. Then a second sample. Then a shared folder of samples. The fix is structural: no access path from MerchNivo into Noveniq's data outside the normal merchant connection, and a written rule that debugging with real data happens only inside Noveniq's own tenant, by people Noveniq authorises, for Noveniq's benefit.

The store's priorities bend. Noveniq starts stocking products or running campaigns because they would make interesting test cases rather than because they make business sense. The fix is Noveniq's own source of truth and checkpoints, which judge the store only on its performance as a store.

One voice dominates the roadmap. The internal store is always available and always articulate, so its requests get built first. The fix is the confirmation step: a request from Noveniq carries no more weight than a request from any other merchant until others share the problem.

Customer communications drift. Noveniq's customers shop at a technology accessories store. Their experience, communications and privacy terms are Noveniq's. If we ever found ourselves wanting to change what Noveniq tells its customers because of MerchNivo, that would be a sign the boundary had slipped.

Learning stays in people's heads. The operator mentions a problem in passing, the MerchNivo team nods, and nothing is written down. Three months later nobody remembers why a threshold was set the way it was. The fix is boring and effective: short written notes, kept with the rest of MerchNivo's product records, so every default can be traced back to the problem that prompted it and the merchants who confirmed it.

Why keep them as separate companies

A reasonable question is why Noveniq and MerchNivo are separate companies at all. Why not make the store part of the software business?

The answer comes from how Oryvelon is designed. Every company has its own brand, customers, revenue model and data, and every company should be able to stand alone. Merging a consumer brand into a software company would mix two very different businesses with different customers, different risks and different economics. It would also make the data boundary much harder to hold.

Keeping them separate brings clear advantages:

• Noveniq's customers deal with a store, not a software experiment. Their data is governed by Noveniq's own policies.
• MerchNivo's customers can trust that it has no special relationship with any one merchant's data, including one owned by the same group.
• Each business is judged on its own results. A strong store does not hide a weak product, or the other way round.
• Either company could be sold, spun out or closed without untangling the other.

This is the same reasoning behind why we don't merge products into one app, applied to a store and a software product.

Lessons for other founders

If you build software for merchants and are considering running a store to learn from, a few suggestions from how we approach it:

• Run it as a real business. Give it real goals and real checkpoints. A fake store teaches fake lessons.
• Write the data boundary down before launch. Decide what crosses and what does not, and put it in both companies' source-of-truth documents.
• Connect through the front door. If the store uses your software, it should connect exactly like any other customer.
• Use synthetic data for development, shaped by what you learned, not copied from the store.
• Treat findings as hypotheses. One store suggests; many merchants confirm.
• Watch for bias. Your own team will forgive things a stranger will not.
• Keep the store's customer communications its own. Never use its customer list to market the software.

Summary

Running a real store as a testbed gives commerce software something interviews and demos cannot: unfiltered problems, real timing and honest verdicts on what is useful. Oryvelon operates Noveniq as a genuine direct-to-consumer business first, and its daily experience informs what MerchNivo builds. What crosses between them is operational learning and product feedback written in plain language, never customer records; if Noveniq uses MerchNivo, it connects like any other merchant into its own isolated tenant, and development relies on synthetic data. Because one store is only one data point, findings from Noveniq are treated as hypotheses and checked with other merchants before they shape the product.

Questions and answers

What does it mean to use a real store as a testbed?

It means operating a genuine online store and using its day-to-day experience to find real problems, test ideas and give feedback on commerce software, while keeping the store's customer data separate.

Does MerchNivo use Noveniq's customer data?

No. Noveniq's customer data belongs to Noveniq. MerchNivo benefits from operational learning and product feedback, not from access to Noveniq's customers.

Can one store validate a commerce software product?

No single store can. A real store is good for finding problems and testing usability quickly, but conclusions are checked with other merchants before they shape the product.

Published by Oryvelon, a digital company builder · About the group · Our companies · More in Commerce & SaaS

NextSubscription plus usage pricing for AI SaaS →