NSM-Workflows

Table of Contents

======>>> Read Doc Format 格式

*https://gtrun.org/custom/workflow.html*

1 2020

2 2019

2.1 Dec

  1. ⚔ INPROCESS 将常用的实践化的 command 写成 pet 的 snippet 并且长期维护

    1. parse zeek log based on articls or PPT :pet:zeek:parse:

      [[snippets]]
        description = "zeek-log:cut zeek log"
        command = "cat <log> | zeek-cut <field>"
        output = ""
      
    2. parse zeek Logs with Vast query language :zeek:Vast:

      [[snippets]]
        description = "zeek|vast: import zeek logs to vast"
        command = "zcat <ZeekLogsPath/*.log.gz | vast import zeek
        output = ""
      
  2. ⚔ INPROCESS 分析细化打标签,细化工作 check list 为 2020 英语公开进度文档流程做准备

  3. ⚔ INPROCESS 为新准备简单快速的文档和实践分析初步

2.2 Nov

  1. ⚔ INPROCESS 根据 SIEM 仓库已有的信息初步将 zeek script 分类 :parse:SIME:Visualization:

  2. 初步学习 open Source SIEM :SIME:Learn:

  3. ✔ DONE 更新 zeek 分析环境逐渐测试 Debian nix 下能够运行。beagle 完成测试,成功 debian 运行 :zeek:Tools:Visualization:

  4. ✔ DONE 将小部分标准化威胁 feed 设定 :zeek:Intel:

  5. ✔ DONE 分类 pcap 在 kafka 下进行测试 json 格式,上传在 github 作为公共分析资料,用 vast 作为 zeek format 和 json 解析查询。 vast parse

  6. ✔ DONE 根据以前文章和 script 在 Tor 进行实践发现问题 :zeek:analyzer:Tor:

2.3 Oct

  1. ✔ DONE 重新解析 zeek http dns 的 top query

  2. 2.2 ✔ DONE 初步 R julia python 探索可视化分析

  3. ✔ DONE 优化部分数据格式

  4. 根据 DDOS 论文更新异常检测数值

  5. ⚔ INPROCESS osquery Notice 初步

2.4 Sep

  1. ✔ DONE update VTHASH & knownhash

    • State “ ✔ DONE” from [2019-10-12 Sat 22:23]
    • add filetypes in knownhash
    • add postgresql(double detectloop) VTlist in VTHASH
    • update jdbc VTHASH
  2. parsing Logstash config file

    • conn
    • dns
    • http
  3. ✔ DONE remove LOgs –> NSM data analysis/ and update NSM data env

    • State “ ✔ DONE” from [2019-10-12 Sat 22:23]
    1. update a part of demo test of Zeek logs Analyzer
  4. ✔ DONE test zeek script & report bugs. update Debian/profile Repo

    • State “ ✔ DONE” from [2019-10-12 Sat 22:23]

    zeek 3.0 test done!

2.5 Aug

  1. ✔ DONE VTHASH Zeek Script API

    • State “ ✔ DONE” from [2019-09-03 Tue 00:39]
  2. ✔ DONE Expire 3 moth( KnownHASH)

    • State “ ✔ DONE” from [2019-09-03 Tue 00:41]
  3. ⚔ ⚔ STARTED CLickhouse VTHASH scheme

  4. ✔ DONE jdbcLogstashConf —> VTHASH database

    • State “ ✔ DONE” from [2019-09-03 Tue 00:42]
  5. ✔ DONE 更新测试 NSM 部件到当前日期版本

    • State “ ✔ DONE” from [2019-09-03 Tue 00:43]

2.6 July

  1. 转移 DNS 的冗余数据和 script,测试 2 个 feed 的布尔值

  2. 对 known-domain/hosts async 机制的编写

  3. 对 Zeek 3.0 R1 新的正则匹配机制算法进行探究和学习,写一些 demo script

    1. ✔ DONE 将不同 dns /conn —> Broker 脚本写法 commit 到 repo
  4. 对 paraglog 的匹配算法进行学习和探究与第三方程序的协调

  5. 对多节点的可视化 sensor 进行管理的相关项目学习和探究

    1. ✔ DONE 完成多节点 zeek topic publish

2.7 Jun

  1. 对公开 playbook feed 格式和其他格式的简要说明

  2. 完善 README

    1. SMTP

    2. FTP

    3. DNS(LOW DNS TTL & DNS to ip changes)

      1. ✔ DONE BRo scirpt test

        • State “ ✔ DONE” from [2019-07-08 Mon 22:01]
    4. Conn (well know IP to suit relationships)

    5. Files( a part of FTP SFTP HTTP/s “YMMV” policy)

2.8 May

  1. 测试 RDP CVE zeek script

  2. 通过公开的 3 本 Playbook 规范 zeek 下检测数据,添加 README

    1. DNS

    2. SSL

    3. conn

    4. http

    5. RDP & VNC

2.9 April

  1. yara rule 初步测试

  2. netflow v5 v9 完全用 silk

  3. http2 vlan bro script 的 2 次处理

  4. 根据 mirte attack 解析 vpn dns 的细化数据

  5. 初步分析 dns 的 silk python

2.10 March

  1. ✔ DONE 修复已经删除部分 zeek/bro 代码问题

    • State “ ✔ DONE” from [2019-04-01 Mon 15:51]
  2. ✔ DONE 更新 intel 和补全基础 intel 问题

    • State “ ✔ DONE” from [2019-04-01 Mon 15:51]
  3. ✔ DONE Zeek Intel to OSINT framework 初步测试

    • State “ ✔ DONE” from [2019-04-01 Mon 15:51]
  4. ✔ DONE 对 NSM data analysis 的测试日志和分化步骤的初步测试

    • State “ ✔ DONE” from [2019-04-01 Mon 15:52]
  5. ✔ DONE 添加以及测试小部分公开的 Osquery secure conf

    • State “ ✔ DONE” from [2019-04-01 Mon 15:54]
  6. ✔ DONE 初步学习 spark 基于 bat 的测试初步的分析从 zeek 接口提取分析

    • State “ ✔ DONE” from [2019-04-01 Mon 16:00]

2.11 Feb

  1. ✔ DONE 重构 SASL 添加 KAFKASecurity Manager

    • State “ ✔ DONE” from [2019-03-01 Fri 16:10]
  2. ✔ DONE 增加 Logstash Betas 配置

    • State “ ✔ DONE” from [2019-03-01 Fri 16:10]
  3. ✔ DONE 更新部分 Intel

    • State “ ✔ DONE” from [2019-03-01 Fri 16:11]
  4. ✔ DONE 增加 NSM 安全分析 repo

    • State “ ✔ DONE” from [2019-03-01 Fri 16:11]

2.12 Jan

  1. ✔ DONE 完成 Kerberos SASL & SSL 测试实践

    • State “ ✔ DONE” from [2019-02-13 Wed 19:03]
  2. ✔ DONE 更新 NSM 主体结构拓扑图

    • State “ ✔ DONE” from [2019-02-13 Wed 19:04]
  3. ✔ DONE 更新 3 个 kafka 的拓扑方案

    • State “ ✔ DONE” from [2019-02-13 Wed 19:04]
  4. ✔ DONE 增加 bro 到 vast 的 Indexing 快速调用到 bat 框架生成流量图像的例子

    • State “ ✔ DONE” from [2019-02-13 Wed 19:05]
  5. ✔ DONE 更新 kafka 和 bro 的加密配置

    • State “ ✔ DONE” from [2019-02-13 Wed 19:07]

2.13 Dec

  1. ✔ DONE 对以往的 Started 任务进行复查到 done 和冗余筛除

    • State “✔ DONE” from “⚔ STARTED” [2019-01-01 Tue 09:43]
  2. ⚔ STARTED 实践 UHH-ISS/honeygrove: A multi-purpose modular honeypot based on Twisted.

  3. ⚔ ⚔ STARTED sasd

  4. 对 bro 源码的 plugin 实践

    1. ✔ DONE 对 Rip 协议的解析和源码编译测试

      • State “✔ DONE” from [2018-12-10 Mon 15:06]
    2. ✔ DONE 检测 fuzzy-hashing 实践

      • State “✔ DONE” from [2018-12-10 Mon 15:07]

      ****

  5. ✔ DONE 对 bro 2.5 –> 2.6 script 测试和解决版本问题

    • State “✔ DONE” from “⚔ STARTED” [2019-01-01 Tue 09:43]
    1. Notice

      1. ✔ DONE DDOS

        • State “✔ DONE” from [2018-12-10 Mon 15:04]
      2. ✔ DONE bronoticecorrelation

        • State “✔ DONE” from [2018-12-10 Mon 15:05]

        ********

  6. ✔ DONE 完成 bro-osquery 端到 bro 的所有问题

    • State “✔ DONE” from [2018-12-10 Mon 15:05]
  7. ✔ DONE 更新迁移 bro-pkg repo & 移除 2.6 废弃特性

    • State “✔ DONE” from [2018-12-10 Mon 15:03]
  8. ✔ DONE 对 samson 文档的细节一些补充和更新,已测试更新到新版本

    • State “✔ DONE” from [2018-12-10 Mon 15:02]
  9. ✔ DONE 实践 pdns(解决 dns 问题)–> jdbc 数据库到 elk 的查询和配置

    • State “✔ DONE” from [2018-12-10 Mon 15:02]
  10. ✔ DONE 修复整体架构的大量细节补充 bro-kafka 的接口和一些脚本的改写

    • State “✔ DONE” from [2019-01-01 Tue 09:44]

2.14 Nov

  1. ✔ DONE 更新 broker 的环境和数据实现 demo 到 bro

    • State “✔ DONE” from [2018-12-10 Mon 14:28]
  2. ✔ DONE 更新 osquery 到 bro-pkg 以及 env 对 readme 的补充

    • State “✔ DONE” from [2018-12-10 Mon 14:16]
  3. ✔ DONE 解决 silk 基础环境的一些细节和补充,支持 silk 3.8 later 版本

    • State “✔ DONE” from [2018-12-10 Mon 14:19]
  4. ✔ DONE Suricata 相关的扩展 evil-box 的搭建测试

    • State “✔ DONE” from [2018-12-10 Mon 14:20]
  5. ⚔ STARTED 解决 silk 新版本 5 个扩展的使用和实践测试(基于论文和新公开的文档)[未 git commit]

    15 号 git,算 11 月进度

    1. Silk-Structure.org~

      1. NewFlow V5 和 3.8 文档的细节对比。Bro 2.6 支持 支持 netflow v5 的解析

      2. ✔ DONE rwcut

        • State “✔ DONE” from [2018-12-10 Mon 14:59]
      3. ✔ DONE rwuniq

        • State “✔ DONE” from [2018-12-10 Mon 14:59]
      4. ✔ DONE rwcount

        • State “✔ DONE” from [2018-12-10 Mon 14:59]
      5. ✔ DONE rwstats

        • State “✔ DONE” from [2018-12-10 Mon 14:59]

2.15 Oct

  1. <2018-10-31 Wed>

    1. ✔ DONE Broker Env shell 完善

      • State “✔ DONE” from [2018-10-31 Wed 22:18]
  2. <2018-10-29 Mon>

    1. Broker-Enabled Communication/Cluster Framework — Bro 2.6-beta2-51 documentation

      Broker 的更换查阅

  3. <2018-10-26 Fri>

    1. ☞ TODO read team ATT TESTING
  4. <2018-10-25 Thu>

    1. ✔ DONE Koild 的搭建测试

      • State “✔ DONE” from [2018-10-25 Thu 17:34]
    2. ✔ DONE Intel IOCs 测试

      • State “✔ DONE” from [2018-10-25 Thu 17:35]
  5. <2018-10-24 Wed>

    1. ⚔ STARTED GitHub - Cyb3rWard0g/ATTACK-Python-Client: Python Script to access ATT&CK content available in STIX via a public TAXII server 对刚开完的 ATT&CKcon 2018 python API 的学习和简单实践
  6. <2018-10-23 Tue>

    1. ⚔ STARTED wazuh/wazuh: Wazuh - Host and endpoint security 参阅 wazuh 移动&改写部分配置到 repo

      • State “⚔ STARTED” from “✔ DONE” [2018-10-25 Thu 17:34]
      • State “✔ DONE” from [2018-10-24 Wed 16:31] 没能测试完
  7. <2018-10-22 Mon>

    1. ✔ DONE Osquery ATT&CK 测试

      • State “✔ DONE” from [2018-10-23 Tue 22:04]
  8. <2018-10-15 Mon>-<2018-10-21 Sun>

    1. ✔ DONE Eventing Framework - osquery

      • State “✔ DONE” from [2018-10-23 Tue 21:35]
    2. ✔ DONE osquery and rsyslog(转发管理日志) 分布式部署

      • State “✔ DONE” from [2018-10-23 Tue 21:50]
    3. ✔ DONE GitHub - kolide/fleet: A flexible control server for osquery fleets

      • State “✔ DONE” from [2018-10-23 Tue 21:55]

      using this tool for incident response and threat hunting scenarios.

    4. ✔ DONE Cyber Wardog Lab: How Hot Is Your Hunt Team? hreat-hunting playbooks 的一些测试

      • State “✔ DONE” from [2018-10-23 Tue 22:04]
    5. ✔ DONE GitHub - mitre/caldera: An automated adversary emulation system

      • State “✔ DONE” from [2018-10-23 Tue 22:04]
    6. ✔ DONE MITRE ATT&CK™ 的基础环境搭建

      • State “✔ DONE” from [2018-10-23 Tue 22:04]
  9. <2018-10-01 Mon>–<2018-10-12 Fri>

    1. ✔ DONE Bro-Osquery 几个 bro 仓库的测试

      • State “✔ DONE” from [2018-10-23 Tue 21:59]
    2. ✔ DONE queries Filtering

      • State “✔ DONE” from [2018-10-22 Mon 22:42]
    3. ✔ DONE CIS Control critical

      • State “✔ DONE” from [2018-10-22 Mon 22:42]
    4. ✔ DONE Analyzers :Bro:

      • State “✔ DONE” from [2018-10-24 Wed 16:37]
    5. ✔ DONE 分类 windows event

      • State “✔ DONE” from [2018-10-24 Wed 16:38]

2.16 Sep

  1. ⚔ STARTED [#A] 更新安全监测的流程表

    1. Hash & md5
  2. ⚔ STARTED Performance testing https://github.com/ncsa/bro-simple-scan

    • 对此脚本的性能测试,总结可优化点 -
  3. ✔ DONE https://github.com/salesforce/hassh

    • State “✔ DONE” from [2018-09-28 Fri 21:07]
      • 根据得到的数值,分类解析到安全数据 List
  4. ✔ DONE 更新 elastic bro 4 个接口下的模板

    • State “✔ DONE” from [2018-09-27 Thu 08:37]
  5. ✔ DONE 学习以及研究 ArangoDB 对安全数据的解析,是否更高效快捷

    • State “✔ DONE” from [2018-09-27 Thu 08:38]
  6. ✔ DONE kibana plugin

    • State “✔ DONE” from [2018-09-27 Thu 09:11]
    1. ✔ DONE https://github.com/JuanCarniglia/area3d%5Fvis

      • State “✔ DONE” from [2018-09-27 Thu 09:05]
    2. ✔ DONE https://github.com/dlumbrer/kbn%5Fnetwork[kbnnetwork]

      • State “✔ DONE” from [2018-09-27 Thu 09:06]
  7. ✔ DONE elastalert-plugin https://github.com/Yelp/elastalert

    • State “✔ DONE” from [2018-09-27 Thu 09:11]
  8. ✔ DONE Clamav 杀毒

    • State “✔ DONE” from “✔ DONE” [2018-09-27 Thu 17:26]
    • State “✔ DONE” from [2018-09-27 Thu 17:26]
  9. ✔ DONE 修复 ISSUE <2018-09-08 Sat>

    • State “✔ DONE” from [2018-09-25 Tue 20:43]

2.17 August

  1. ⚔ STARTED 完成 snort suricata rules 之间转换的 demo<2018-08-31 Fri>

  2. ⚔ STARTED 对已有的安全标标准来分化结构已有和未来的数据格式<基于美国网络安装标准和主流安全指南><2018-08-31 Fri>

  3. ✔ DONE Bro Script <2018-08-27 Mon> :补 8 月工作进度:

    • State “✔ DONE” from “⚔ STARTED” [2018-08-31 Fri 01:46]

    *审计其他接口开源的 Bro Script 放入默认 hardened NSM 仓库*

  4. ⚔ STARTED Broccoli Data analysis :补 8 月工作进度:

    1. Python 接口的数据解析 3 个接口下的 <2018-08-29 Wed>
  5. Suricata

    1. ⚔ STARTED https://suricata.readthedocs.io/en/suricata-4.0.5/ 从官方文档中细化结构 checklist<2018-08-24 Fri>

      https://suricata-update.readthedocs.io/en/latest/

    2. ✔ DONE Signature = malware detection

      • State “✔ DONE” from [2018-08-27 Mon 17:21]
    3. ✔ DONE application detection with suricata

      • State “✔ DONE” from [2018-08-27 Mon 17:21]
    4. ✔ DONE Pccap Analysis

      • State “✔ DONE” from [2018-08-27 Mon 17:23]
  6. Con

    1. ☞ TODO [#A] https://github.com/mattifestation/BHUSA2018%5FSysmon<2018-08-10 Fri>

    2. ☞ TODO https://www.trustedsec.com/2018/05/art%5Fof%5Fkerberoast/ [Kerberoast

2.18 July

  1. developing

    1. 应用识别改进方案[0/1]

      1. [X] [] 整合重复性的脚本

      2. ⚔ STARTED python-Broccoli + redis

        1. ✔ DONE 读取 redis 已有的数据发送到 bro <2018-07-27 Fri -27h>

          • State “✔ DONE” from [2018-07-28 Sat 12:54]
        2. ☞ TODO 自动读取更新数据库

  2. <2018-07-24 Tue>

    1. ✔ DONE Debugger & debugging with log for Bro

      • State “✔ DONE” from [2018-07-28 Sat 12:15]
    2. ✔ DONE Test Bro Script

      • State “✔ DONE” from [2018-07-28 Sat 12:15]
  3. <2018-07-19 Thu>

    1. detect System

      1. 2

        定义一个静态 global static system OS

        • http protocol
          • 抓取 uri post body 的关于软件版本以及设备信息
            • 抓取到的设备信息更新到 global static system OS
      2. ✔ DONE 1 [舍弃]

        • State “✔ DONE” from [2018-07-19 Thu 15:00]

        appkeywordlist

        • http-protocol
          If (app_keyword in app_keyword_list)
          {
              if (user-agent in mobile_keyword_list)
                      {
                          software_type=mobile_type;
                      }
                  else
                      nmap -O <target_ip> = res_OS;
                      res_OS=c
                 }
        
  4. <2018-07-01 Sun>-<2018-07-14 Sat>

    1. Broccoli

2.19 Jun

  1. <2018-06-28 Thu>

    1. ✔ DONE compare hash file http://www.malware-domains.com/files/

      • State “✔ DONE” from [2018-07-04 Wed 13:24]
  2. <2018-06-27 Wed>

    1. ✔ DONE Broccoli 初步编写

      • State “✔ DONE” from “⚔ STARTED” [2018-07-26 Thu 13:25]
    2. ⚔ STARTED review open source of the suricata rules which is can able to coalesce into NSM.

    3. ✔ DONE Suricata signature matches[33]

      • State “✔ DONE” from “⚔ STARTED” [2018-07-26 Thu 13:25]
      1. ✔ DONE SSL/TLS traffic

      2. ✔ DONE Ransomware detect suricata

        • State “✔ DONE” from “⚔ STARTED” [2018-07-26 Thu 13:25]
      3. ✔ DONE metron-bro-plugin-kafka example 2

  3. <2018-06-26 Tue>

    1. ⚔ STARTED open protocol of the tcp for Bro script

    2. ✔ DONE Using SASL with librdkafka

      • State “✔ DONE” from [2018-06-27 Wed 16:39]
    3. 思考 http://www.malware-domains.com/ 的数据动态爬取更新到数据库

      1. ✔ DONE dns 加入 bro script

        • State “✔ DONE” from [2018-06-27 Wed 08:20]
  4. <2018-06-23 Sat>-<2018-06-25 Mon>

    1. 测试思考采用的流程

      1. Bro-broccoli[连接 bro]–>bat[解析学习数据,生成深度学习图片]——>pyspark 处理日志传递到 kafka 的数据
    2. 测试开源框架

      1. ✔ DONE evebox https://evebox.org/

        • State “✔ DONE” from [2018-06-26 Tue 18:05]
      2. ✔ DONE https://github.com/StamusNetworks [确定采用部分配置和代码]

        • State “✔ DONE” from [2018-06-26 Tue 18:07]
    3. 测试脚本组件

      1. ✔ DONE http://justinazoff.github.io/netflow-indexer/configuration.html#example-configuration-files

        • State “✔ DONE” from [2018-06-26 Tue 18:06]
      2. ✔ DONE https://github.com/JustinAzoff/flow-indexer

        • State “✔ DONE” from [2018-06-26 Tue 18:09]
  5. ✔ DONE <2018-06-21 Thu>

    • State “✔ DONE” from “⚔ STARTED” [2018-06-26 Tue 18:25]
    1. ⚔ STARTED Basci Suricata rules

      1. Message and Content
      2. Header
      3. Metadata and PCRE
    2. ✔ DONE Bro script -files-exploitkit [developing] :FILES:

      • State “✔ DONE” from [2018-06-21 Thu 20:52]

      https://github.com/sooshie/bro-scripts/tree/master/exploitkit

    3. ✔ DONE Filter-Bro-protocol-files

      • State “✔ DONE” from [2018-06-26 Tue 18:23]
    4. ✔ DONE Bro-protocol-dns–> detect-dynamic dns domains :DNS:

      • State “✔ DONE” from “⚔ STARTED” [2018-06-26 Tue 18:25]
  6. <2018-06-11 Mon>-<2018-06-20 Wed>

    1. 修复几个 Bug

    2. 浏览设计框架

    3. 补充 Elastic bro http.log format

  7. <2018-06 -01 Fri>-<2018-06-10 Sun>

    1. Bro Script Application

3 Project

3.1 Zeek

  1. Framework

    1. ✔ DONE Intel Framework <2018-10-24 Wed>

      • State “✔ DONE” from [2018-10-24 Wed 16:36]
    2. ✔ DONE Input Framework<2018-07-26 Thu>

      • State “✔ DONE” from [2018-07-28 Sat 12:17]
    3. ✔ DONE Summary Statistics <2018-07-21 Sat>

      • State “✔ DONE” from [2018-07-28 Sat 12:17]
  2. Protocol

    1. ✔ DONE Disable Analyzers <2018-10-24 Wed>

      • State “✔ DONE” from [2018-10-24 Wed 16:38]
    2. ✔ DONE base/protocols/conn/thresholds <2018-07-20 Fri>

      • State “✔ DONE” from [2018-07-28 Sat 12:21]
    3. ✔ DONE sshauthfailed<2018-07-29 Sun>

      • State “✔ DONE” from [2018-07-29 Sun 22:04]
  3. Filtering

    1. ✔ DONE without Referrers HTTP (method)<2018-07-13 Fri> :HTTP:

      • State “✔ DONE” from [2018-07-28 Sat 12:23]
    2. ✔ DONE Filter DNS Logs without id.resph transid qcalssname record etc<2018-07-09 Mon> :DNS:

      • State “✔ DONE” from [2018-07-28 Sat 12:27]

      ******** ************

      -

3.2 Suricata

3.3 Silk

Avatar
Guangtao Zhang
Threat Intelligence Analyst

My research interests include IDS, Monitoring system and Data analysis

Related