Using a workflow engine, state machine engine or rolling my own?

I can't agree more with above AMS' answer, and one more thing i want to add is for most scenarios, using workflow/rule engine is overkill and unnecessary. KISS（Keep It Simple and Stupid）is always the best choice. and Occam's Razor also says "Entities should not be multiplied unnecessarily"

Per my own working experience in Alibaba, most workflow/rule-engine equipped applications are putting the maintainance into nightmare, and people comes to project later will appreciate you if you use simplified impl rather than blindly choose workflow/rule engine.

So, is there a guideline telling when to use workflow or not? frankly i don't know, but what i know is it's definitely not we should use workflow whenever the biz logic is in a flow. because if you will, every biz logic can be presented in a flow chart.

Lastly, one of my most correct thing i did last year is redesign an application to replace Drools by groovy script, which makes the whole system much more straightforward, simpler and faster.

The main value of a Workflow engine is that it makes it possible to customize the flows through some workflow definition DSL. If you don't need to allow users to define their own arbitrary workflows, they you are better off just building your own.

Also workflow engines usually give you the ability to define business transactions & rules that are very long running. For example, you can have a workflow for authorizing purchase orders, where the first step is to enter some information about what needs to be purchased, then you have rules along the lines if the purchase is for less $100 okay it right away, if its between $100 & $2000 line manager can okay, if it's more then send it to some one else for approval ... etc. These types of business rules tend to change over the years as the amounts get increased, or the business policies for a company change. So it makes sense to use a workflow engine in those scenarios. Other good examples of complex business transactions that can benefit from a workflow engine are making an insurance claim, authorizing a loan or a mortgage, assessing a credit application from a customer ... etc. These business transactions tend to go through several people / departments and take several hours to days or weeks to complete.

Rule engines are good for extracting complex but changing rules from an application. Lets say you are an online retailer that ships to customer in the USA, Canada, UK, Germany, and France. You are required to charge taxes on the products you sell on your online shop but the rules for calculating taxes are different from country to country and from province to province within a country. Also some things are exempt from tax in one province but not in other provinces. Rule engines are great for these types of complex business rules that can change whenever the government changes their tax policy. Rules engines can give you an answer right way you just have to go to the rule engine say I want to run rule #10 and here are the inputs for rule #10 x,y,z and you get back an answer.

Main differences between a rule engine and a workflow engine, is that rule engine does not track the state of the transaction, it should be stateless working only on the inputs you provide it. Workflow engine is statefull, it must know what current state is the workflow in and must save that state to a database. Workflow engines also wait for input from external sources such as people or systems.

From what you are describing about your app I would just write some groovy classes to compute the next state of a ticket and make sure that the class is well documented and easy to update in a few years. I think rule engines and workflow engines are overkill for your situation, the amount of time it would take you to set them up and use them is much bigger that it would take you to write the code in groovy. If over time you discover you need the complexity of rule engines and workflow engines, I would pay the price then rather than now, keeping it simple is always the best choice.

'State machine' is common design pattern, so what drools actually gives you? I personally value drools for its 'query language', this is what makes it shine. You practically have something like 'SQL to query objects from your heap'. Just like SQL gives you 'declarative' way of programming, drools when block describes when to start state transition in declarative way. Drools was designed to be statefull by default and state is all facts (POJOs) which were inserted into drools session.

Let me propose you simple usecase. You have to write application for cellular phone company to manage phone calls:
If caller 1 is calling to callee 2 and he is not 'busy' at that MOMENT, connect them.
If callee is busy, continue calling for 7 seconds and if the callee dismiss his original call DURING that period of time, connect them at once.
if callee will not disconnect during 7 seconds, drop the caller with the message 'callee is busy'.

Simple triple-if statement business method quickly became quite complex and error prone technical task. I imagine background Timers' I've seen a lot 5 to 10 years ago or some newer, like ScheduledThreadPoolExecutor. And what if state changed DURING scheduled delay? Will you still wait until the end to recalculate the condition? If such condition is relatively often in your application or the period is relatively long will you keep 'context' in the memory? You would need to keep track of Futures and cancel them or use some BlockingQueue. One would need to maintain queue for each person for such cases because each person can be potentially called by somebody. Traditional OOP says 'you should keep behavior attached to your domain entities'. With this approach you'll start clutter your business entities with quite complex technical stuff even if you would use some patterns to simplify (encapsulate) complexity your 'state machine' start to spread between multiple components, it will be even worse if you'll use 'layered style' because you will start to produce data structures for the state stuff, like Map<Callee, BlockingQueue<Caller>>. Next day your customer come to you and say, 'hey, I have another simple use case for you around phone calls'.

Drools solves such problems 'naturally' because it uses totally different approach. It keeps track of all objects in the working memory (it keeps track of the rule condition) and when time comes, drools is just able to say whether your rule is eligible to be executed or not (rete algorithm back to 1979). If state changes, drools re-evaluates when condition for each rule in efficient way and put or remove respective rules from 'agenda' (execution queue). All objects you've inserted into 'working memory' form a 'state' which any rule can relay on. You can find some diagrams and tests for the usecase described above and actual drools rule implementation here

Use right tools for your tasks. If you need for loop on collection of entities with State object inside with 3-5 fields, don't put 'state machine' in the title. If you really face problems like 'continuous behavior change' or complex cause-effect dependencies between events in your system, then drools is good and time-proven open source rete algorithm implementation. Don't try to use everything advertised, dig into details, understand what suit your needs.