Figure 1
Illustration of the research design process with references to detailed content and result of each stage.
Figure 2
Screenshot of the participant user interface.
Note: The figure has been annotated with numbers in red, showing the location of the opinion input field (1), agreement slider (2), user number (3), opinion list (4), opinion history link (5), facilitator message (6), assessment file links (7), and draft report (8).
Figure 3
Process views of the initial method (left) and the developed MUAFT v2 method (right).
Table 1
Footprints from the assessed photonic radar technology.
| INFLUENCED FACTOR | FOOTPRINT |
|---|---|
| Doctrine | MIMO might imply changed doctrines and tactics regarding operational use of the sensor chain. |
| Organization | The unmanned systems and the new sensor chain will have to be incorporated in the organisation. |
| Training | Commanders need education to understand the usability of photonic radars to exploit new capabilities due to Low Probability of Intercept (LPI) |
| Materiel | New unmanned platforms to carry photonic radars and new communication links, possibly satellites, are needed. |
| Leadership | A new kind of leadership may have to be developed to communicate with machines. |
| Personnel | Fewer personnel will be required on the introduction of unmanned platforms. |
| Facilities | Facilities for maintenance of unmanned platforms and photonic radars are needed. |
| Interoperability | Military standards and standardization agreements must be followed. |
[i] Note: The results from the evaluation of the updated method were considered insufficient to illustrate this step.
Figure 4
Survey responses to questions with interval variables in the reference and post-development surveys.
Figure 5
Survey responses from participants in assessments of the military utility using the MUAFT method before and after the reported development.
Note: “Don’t know” responses are reported separately on the right-hand side.
Figure 6
Survey responses to the additional questions sent to participants in the post-development experiment.
Note: “Don’t know” responses are reported separately on the right-hand side.
Table 2
The structure used to capture experiences from the research team.
| MAJOR PROCESS ACTIVITIES | CONCERNS DISCUSSED |
|---|---|
| Choose advocate | Simultaneously being the facilitator? |
| Preparation | Providing technical support; providing education to expert participants; maintaining the application; entering the first part of the memo; selecting and inviting expert participants; offering incentives for participation; administering participants with assured anonymity. |
| Asynchronous web-based assessment | Dividing the review into appropriate steps; the facilitator’s dialogue with the participants, including formulating messages, updates, and their frequency; management of those invited who do not participate from the outset; management of additional participants during review period; length of review period and possible end criteria; tool improvements towards increased validity, increased anonymity and robustness/reliability. |
| Conclusions and recommendations | Whether conclusions and recommendations should form a separate step in the process; design of voting procedure for enhanced anonymity. |
| Term | Definition |
|---|---|
| Conceptualization | Ideas; formulation or evolution of overarching research goals and aims |
| Methodology | Development or design of methodology; creation of models |
| Software | Programming, software development; designing computer programs; implementation of the computer code and supporting algorithms; testing of existing code components |
| Validation | Verification, whether as a part of the activity or separate, of the overall replication/reproducibility of results/experiments and other research outputs |
| Formal analysis | Application of statistical, mathematical, computational, or other formal techniques to analyze or synthesize study data |
| Investigation | Conducting a research and investigation process, specifically performing the experiments, or data/evidence collection |
| Resources | Provision of study materials, reagents, materials, patients, laboratory samples, animals, instrumentation, computing resources, or other analysis tools |
| Data Curation | Management activities to annotate (produce metadata), scrub data and maintain research data (including software code, where it is necessary for interpreting the data itself) for initial use and later reuse |
| Writing – Original Draft | Preparation, creation and/or presentation of the published work, specifically writing the initial draft (including substantive translation) |
| Writing – Review & Editing | Preparation, creation and/or presentation of the published work by those from the original research group, specifically critical review, commentary or revision – including pre-or postpublication stages |
| Visualization | Preparation, creation and/or presentation of the published work, specifically visualization/data presentation |
| Supervision | Oversight and leadership responsibility for the research activity planning and execution, including mentorship external to the core team |
| Project administration | Management and coordination responsibility for the research activity planning and execution |
| Funding acquisition | Acquisition of the financial support for the project leading to this publication |