Custom Models + LLMs: A Proposed Pipeline for Decision Support and Intelligence Reporting

The Large Language Model (LLM) is one of the most popular topics in Artificial Intelligence. Computer-generated human-like text is nothing short of amazing. The rise of the LLM has brought with it prompt engineering which is the practice of refining instructive text that an LLM uses to produce a suitable response. This practice is rather useful when producing human-readable insights for various project requirements, especially when the prompt is fed with data obtained from Machine Learning and Deep Learning models scoped to a specific problem area. This article proposes a simple pipeline to combine custom trained models and LLMs, applying this architecture to Defence and Security projects, in order to obtain trusted intelligence reports containing recommended decisions.

Overview of Pipeline

The proposed pipeline is one that accepts reliable results from custom trained models to extract relevant details complemented by system environment data. These details are used to construct a robust prompt to be passed to an LLM in order to receive human readable insights. The main idea is to architect your system in such a way that you define what modules are relevant to your problem such as suspicious military base video activity during unauthorised hours, anomalous vessel activity near restricted waters or rapidly moving unidentified aircraft. The pipeline requires you to gather data for these modules and train appropriate classification or predictive models. The results of these models alongside generic system environment data are implemented to save only relevant information for suggested actions in a JSON object. This JSON object is used to construct a scenario-specific prompt to pass to an LLM that generates human-readable insights in any format of your liking. This format could be a short summary or a more detailed report.

Custom Models

The proposed pipeline requires training models that meet the specific requirements for a project. In the context of Defence and Security, this step may take the form of training an object detection network that analyses military base footage and detects suspicious activity or items. Another application could be to train a decision tree that determines (classifies) an appropriate military action for environment inputs with the aid of historical actions taken for the environment inputs of those past data points. The key point here is to have one or multiple models that produce results you can trust for your precise problem area. This step requires rigorous model evaluation, taking into account metrics such as accuracy, precision, recall, F1-score, mean average precision (mAP) and intersection over union (IoU) in order to save the top-performing model.

Once one or more satisfactory models have been trained, new environment data may be passed to these models to receive appropriate classifications and predictions for various modules such as the detection of potentially dangerous objects and the suggestion of appropriate follow-up actions as outlined in the aforementioned examples. These outputs, typically categorical or numeric, serve as the inputs for the next step of the pipeline, enriching the prompt for the LLM to process and generate scenario-specific insights.

Extracting Details with the System Environment and Custom Model Outputs

With the output of the custom trained models, we may reliably make predictions based on incoming environment data. The output of this model allows us to standardise a JSON object to capture only the relevant details. Let us say that our object detection model predicts that there are two persons near a box on footage at a military base during unauthorised hours. Additionally, let us say that our classification model determines that the appropriate follow-up action for this scenario is to report these findings to the higher-ups.

We may code a script to extract only the pertinent details of these model outputs to build a JSON object. This JSON object may indicate that the current system environment is in a state of PEACE. Moreover, the JSON object may save the detected objects with their classes (person and box) and their locations at a precise timestamp in the video footage for the provided military base in a specific room at a specific hour on a specific day. In this case, our system detects that these objects were detected during unauthorised hours. The JSON object may also include the associated feasible but high-risk suggestion of type REPORT_TO_HIGHER_UPS.

Construct the Prompt

With the aforementioned JSON object, we may construct a specific prompt for the LLM. This result may be achieved by coding a script that uses a base prompt template with regex pattern matching to fill in the appropriate outputs of the custom models and details from the system environment.

Here is an example prompt:

“Current system environment is in a state of PEACE. Software detects two persons and an unnamed box during unauthorised hours of the morning at Defence Facility X2026, specifically at 02:02h on January 11, 2026 in Logistics Bay (Zone B-2). Software suggests a feasible, high-risk action to report the detected activity to the higher-ups.”

Pass Outputs to LLM to Receive Human-Readable Insights

Using this constructed prompt, the LLM is able to generate detailed reports about the suggested action, citing specific reasons. It is important to note that, in this pipeline, the LLM itself is not what makes the suggestion. Instead it uses upstream output from the custom trained models as well as generic system environment data present in the prompt to explain the suggestions. The LLM simply accepts this contextual technical data to generate a report that is easily interpretable for humans.

Does the LLM Model Need Finetuning?

Finetuning is an AI technique where a pretrained model is trained with an additional dataset from a particular problem-domain so that the model is better adapted to produce results within that domain. For example, Gemma 3 is a pre-trained model provided by Google. It is trained with a massive dataset across various sources and topics. This broad training data means that it is able to generate generic military-related responses. However, it does not inherently know company-specific data. In order to achieve this task, an organisation may provide a dataset of questions related to internal military scenarios with their accompanying action-based responses. By fine-tuning the Gemma 3 model with this dataset, the model may provide more appropriate responses, citing past scenarios within the organisation. When fine-tuning models with private company data, it is recommended to do so in an on-premise environment or private cloud.

It is important to note however that fine-tuning is time-consuming and not always necessary. For simple reports, especially those generated with a prompt that already contains sufficient data for the given scenario, an out-of-the-box model might produce satisfactory reports. However, there are projects that require lengthy organisational context. In that case, it might not be favourable to pass a long prompt that attempts to capture extensive past information. Instead it is preferable to fine-tune the model with organisational historical data and then to pass a shorter prompt for the specific scenario at hand.

We are excited to see how you combine custom trained models and LLMs to generate intelligence reports for your Defence and Security decision-making projects. Let us know how it goes!