aya/myos
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

import files

Browse Source
This commit is contained in:
Yann Autissier
2021-02-09 17:05:00 +01:00
parent f5c4576411
commit 44a6d37ba5
425 changed files with 23195 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ansible/roles/disks/.gitrepo
+12
View File
@@ -0,0 +1,12 @@
; DO NOT EDIT (unless you know what you are doing)
;
; This subdirectory is a git "subrepo", and this file is maintained by the
; git-subrepo command. See https://github.com/git-commands/git-subrepo#readme
;
[subrepo]
remote = ssh://git@github.com/1001Pharmacies/ansible-disks
branch = master
commit = c0ac6978d715b461fbf20aca719cd5196bc60645
parent = d01cccd9bab3a63d60ba251e3719767635ccd5d2
method = merge
cmdver = 0.4.0
ansible/roles/disks/LICENSE
+21
View File
@@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) 2014 Wizcorp
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
ansible/roles/disks/README.md
+77
View File
@@ -0,0 +1,77 @@
Disk
====
This role allows you to format extra disks and attach them to different mount points.
You can use it to move the data of different services to another disk.
Configuration
-------------
### Inventory
Because the configuration for additional disks must be stored using the YAML
syntax, you have to write it in a `group_vars` directory.
```yaml
# inventory/group_vars/GROUP_NAME
disks_additional_disks:
- disk: /dev/sdb
fstype: ext4
mount_options: defaults
mount: /data
user: www-data
group: www-data
disable_periodic_fsck: false
- disk: /dev/nvme0n1
part: /dev/nvme0n1p1
fstype: xfs
mount_options: defaults,noatime
mount: /data2
- device_name: /dev/sdf
fstype: ext4
mount_options: defaults
mount: /data
- disk: nfs-host:/nfs/export
fstype: nfs
mount_options: defaults,noatime
mount: /mnt/nfs
```
* `disk` is the device, you want to mount.
* `part` is the first partition name. If not specified, `1` will be appended to the disk name.
* `fstype` allows you to choose the filesystem to use with the new disk.
* `mount_options` allows you to specify custom mount options.
* `mount` is the directory where the new disk should be mounted.
* `user` sets owner of the mount directory (default: `root`).
* `group` sets group of the mount directory (default: `root`).
* `disable_periodic_fsck` deactivates the periodic ext3/4 filesystem check for the new disk.
You can add:
* `disks_package_use` is the required package manager module to use (yum, apt, etc). The default 'auto' will use existing facts or try to autodetect it.
The following filesystems are currently supported:
- [btrfs](http://en.wikipedia.org/wiki/BTRFS) *
- [ext2](http://en.wikipedia.org/wiki/Ext2)
- [ext3](http://en.wikipedia.org/wiki/Ext3)
- [ext4](http://en.wikipedia.org/wiki/Ext4)
- [nfs](http://en.wikipedia.org/wiki/Network_File_System) *
- [xfs](http://en.wikipedia.org/wiki/XFS) *
*) Note: To use these filesystems you have to define and install additional software packages. Please estimate the right package names for your operating system.
```yaml
# inventory/group_vars/GROUP_NAME
disks_additional_packages:
- xfsprogs # package for mkfs.xfs on RedHat / Ubuntu
- btrfs-progs # package for mkfs.btrfs on CentOS / Debian
disks_additional_services:
- rpc.statd # start rpc.statd service for nfs
```
How it works
------------
It uses `sfdisk` to partition the disk with a single primary partition spanning the entire disk.
The specified filesystem will then be created with `mkfs`.
Finally the new partition will be mounted to the specified mount path.
ansible/roles/disks/defaults/main.yml
+8
View File
@@ -0,0 +1,8 @@
---
# Aditional disks that need to be formated and mounted.
# See README for syntax and usage.
disks_additional_disks: []
disks_additional_packages: []
disks_additional_services: []
disks_discover_aws_nvme_ebs: False
disks_package_use: auto
ansible/roles/disks/handlers/main.yml
+21
View File
@@ -0,0 +1,21 @@
---
# file: handlers/main.yml
- name: restart services
with_together:
- '{{ disks_additional_disks }}'
- '{{ disks_additional_disks_handler_notify.results }}'
service:
name: "{{item.0.service}}"
state: restarted
when: item.1.changed and item.0.service is defined
- name: restart services - nfs
with_together:
- '{{ disks_additional_disks }}'
- '{{ disks_additional_disks_nfs_handler_notify.results }}'
service:
name: "{{item.0.service}}"
state: restarted
when: item.1.changed and item.0.service is defined
ansible/roles/disks/library/disks_ebs_config.py
+182
View File
@@ -0,0 +1,182 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
from ctypes import *
from fcntl import ioctl
from pathlib import Path
import json
import os
import subprocess
from ansible.module_utils.basic import *
module = AnsibleModule(argument_spec=dict(
config=dict(required=True, type='list'),
))
NVME_ADMIN_IDENTIFY = 0x06
NVME_IOCTL_ADMIN_CMD = 0xC0484E41
AMZN_NVME_VID = 0x1D0F
AMZN_NVME_EBS_MN = "Amazon Elastic Block Store"
class nvme_admin_command(Structure):
_pack_ = 1
_fields_ = [("opcode", c_uint8), # op code
("flags", c_uint8), # fused operation
("cid", c_uint16), # command id
("nsid", c_uint32), # namespace id
("reserved0", c_uint64),
("mptr", c_uint64), # metadata pointer
("addr", c_uint64), # data pointer
("mlen", c_uint32), # metadata length
("alen", c_uint32), # data length
("cdw10", c_uint32),
("cdw11", c_uint32),
("cdw12", c_uint32),
("cdw13", c_uint32),
("cdw14", c_uint32),
("cdw15", c_uint32),
("reserved1", c_uint64)]
class nvme_identify_controller_amzn_vs(Structure):
_pack_ = 1
_fields_ = [("bdev", c_char * 32), # block device name
("reserved0", c_char * (1024 - 32))]
class nvme_identify_controller_psd(Structure):
_pack_ = 1
_fields_ = [("mp", c_uint16), # maximum power
("reserved0", c_uint16),
("enlat", c_uint32), # entry latency
("exlat", c_uint32), # exit latency
("rrt", c_uint8), # relative read throughput
("rrl", c_uint8), # relative read latency
("rwt", c_uint8), # relative write throughput
("rwl", c_uint8), # relative write latency
("reserved1", c_char * 16)]
class nvme_identify_controller(Structure):
_pack_ = 1
_fields_ = [("vid", c_uint16), # PCI Vendor ID
("ssvid", c_uint16), # PCI Subsystem Vendor ID
("sn", c_char * 20), # Serial Number
("mn", c_char * 40), # Module Number
("fr", c_char * 8), # Firmware Revision
("rab", c_uint8), # Recommend Arbitration Burst
("ieee", c_uint8 * 3), # IEEE OUI Identifier
("mic", c_uint8), # Multi-Interface Capabilities
("mdts", c_uint8), # Maximum Data Transfer Size
("reserved0", c_uint8 * (256 - 78)),
("oacs", c_uint16), # Optional Admin Command Support
("acl", c_uint8), # Abort Command Limit
("aerl", c_uint8), # Asynchronous Event Request Limit
("frmw", c_uint8), # Firmware Updates
("lpa", c_uint8), # Log Page Attributes
("elpe", c_uint8), # Error Log Page Entries
("npss", c_uint8), # Number of Power States Support
("avscc", c_uint8), # Admin Vendor Specific Command Configuration
("reserved1", c_uint8 * (512 - 265)),
("sqes", c_uint8), # Submission Queue Entry Size
("cqes", c_uint8), # Completion Queue Entry Size
("reserved2", c_uint16),
("nn", c_uint32), # Number of Namespaces
("oncs", c_uint16), # Optional NVM Command Support
("fuses", c_uint16), # Fused Operation Support
("fna", c_uint8), # Format NVM Attributes
("vwc", c_uint8), # Volatile Write Cache
("awun", c_uint16), # Atomic Write Unit Normal
("awupf", c_uint16), # Atomic Write Unit Power Fail
("nvscc", c_uint8), # NVM Vendor Specific Command Configuration
("reserved3", c_uint8 * (704 - 531)),
("reserved4", c_uint8 * (2048 - 704)),
("psd", nvme_identify_controller_psd * 32), # Power State Descriptor
("vs", nvme_identify_controller_amzn_vs)] # Vendor Specific
class ebs_nvme_device:
def __init__(self, device):
self.device = device
self.ctrl_identify()
def _nvme_ioctl(self, id_response, id_len):
admin_cmd = nvme_admin_command(opcode = NVME_ADMIN_IDENTIFY,
addr = id_response,
alen = id_len,
cdw10 = 1)
with open(self.device, "w") as nvme:
ioctl(nvme, NVME_IOCTL_ADMIN_CMD, admin_cmd)
def ctrl_identify(self):
self.id_ctrl = nvme_identify_controller()
self._nvme_ioctl(addressof(self.id_ctrl), sizeof(self.id_ctrl))
def is_ebs(self):
if self.id_ctrl.vid != AMZN_NVME_VID:
return False
if self.id_ctrl.mn.strip() != AMZN_NVME_EBS_MN:
return False
return True
def get_volume_id(self):
vol = self.id_ctrl.sn.decode('utf-8')
if vol.startswith("vol") and vol[3] != "-":
vol = "vol-" + vol[3:]
return vol.strip()
def get_block_device(self, stripped=False):
dev = self.id_ctrl.vs.bdev.decode('utf-8')
if stripped and dev.startswith("/dev/"):
dev = dev[5:]
return dev.strip()
def update_disk(disk, mapping):
if 'device_name' not in disk:
return disk
device_name = disk['device_name'][5:]
if device_name not in mapping:
return disk
volume_id = mapping[device_name]
link_path = '/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol%s' % volume_id[4:]
resolved = str(Path(link_path).resolve())
new_disk = dict(disk)
new_disk['disk'] = resolved
new_disk['part'] = '%sp1' % resolved
return new_disk
def main():
src_config = module.params['config']
lsblkOutput = subprocess.check_output(['lsblk', '-J'])
lsblk = json.loads(lsblkOutput.decode('utf-8'))
mapping = {}
for blockdevice in lsblk['blockdevices']:
try:
dev = ebs_nvme_device('/dev/%s' % blockdevice['name'])
except OSError:
continue
except IOError:
continue
if dev.is_ebs():
continue
mapping[dev.get_block_device()] = dev.get_volume_id()
new_config = [
update_disk(disk, mapping) for disk in src_config
]
facts = {'blockDeviceMapping': mapping, 'config': new_config, 'source_config': src_config}
result = {"changed": False, "ansible_facts": facts}
module.exit_json(**result)
main()
ansible/roles/disks/meta/main.yml
+21
View File
@@ -0,0 +1,21 @@
galaxy_info:
author: Emilien Kenler <ekenler@wizcorp.jp>
description: This role allows setting up extra disks and their mount points
company: Wizcorp K.K.
license: MIT
min_ansible_version: 2.0.0
platforms:
- name: EL
versions:
- 6
- 7
- name: Debian
versions:
- wheezy
- jessie
- name: Ubuntu
versions:
- all
categories:
- system
dependencies: []
ansible/roles/disks/tasks/main.yml
+173
View File
@@ -0,0 +1,173 @@
- name: 'Install Python PIP'
package: >
name=py3-pip
state=present
when: ansible_os_family|lower == "alpine"
- name: 'Install Python PIP'
package: >
name=python-pip
state=present
when: ansible_os_family|lower != "alpine"
- name: 'Install python-pathlib'
pip: >
name=pathlib
state=present
- name: "Discover NVMe EBS"
disks_ebs_config:
config: "{{ disks_additional_disks }}"
register: __disks_ebs_config
when: disks_discover_aws_nvme_ebs | default(True) | bool
- set_fact:
disks_additional_disks: "{{ disks_additional_disks|defaut([]) + __disks_ebs_config['ansible_facts']['config'] }}"
when: __disks_ebs_config is defined and 'ansible_facts' in __disks_ebs_config
- name: "Install parted"
package:
name: parted
state: present
use: '{{ disks_package_use }}'
when: disks_additional_disks
tags: ['disks', 'pkgs']
- name: "Install additional fs progs"
package:
name: "{{ item }}"
state: present
with_items: "{{ disks_additional_packages|default([]) }}"
when: disks_additional_packages is defined
tags: ['disks', 'pkgs']
- name: disks - start additional services
service:
name: "{{item}}"
enabled: yes
state: started
with_items: "{{ disks_additional_services|default([]) }}"
tags: ['disks', 'pkgs']
- name: "Get disk alignment for disks"
shell: |
if
[[ -e /sys/block/{{ item.disk | basename }}/queue/optimal_io_size && -e /sys/block/{{ item.disk | basename }}/alignment_offset && -e /sys/block/{{ item.disk | basename }}/queue/physical_block_size ]];
then
echo $[$(( ($(cat /sys/block/{{ item.disk | basename }}/queue/optimal_io_size) + $(cat /sys/block/{{ item.disk | basename }}/alignment_offset)) / $(cat /sys/block/{{ item.disk | basename }}/queue/physical_block_size) )) | 2048];
else
echo 2048;
fi
args:
creates: '{{ item.part | default(item.disk + "1") }}'
executable: '/bin/bash'
with_items: '{{ disks_additional_disks }}'
register: disks_offset
tags: ['disks']
- name: "Ensure the disk exists"
stat:
path: '{{ item.disk }}'
with_items: '{{ disks_additional_disks }}'
register: disks_stat
changed_when: False
tags: ['disks']
- name: "Partition additional disks"
shell: |
if
[ -b {{ item.disk }} ]
then
[ -b {{ item.part | default(item.disk + "1") }} ] || parted -a optimal --script "{{ item.disk }}" mklabel gpt mkpart primary {{ disks_offset.stdout|default("2048") }}s 100% && sleep 5 && partprobe {{ item.disk }}; sleep 5
fi
args:
creates: '{{ item.part | default(item.disk + "1") }}'
executable: '/bin/bash'
with_items: '{{ disks_additional_disks }}'
tags: ['disks']
- name: "Create filesystem on the first partition"
filesystem:
dev: '{{ item.0.part | default(item.0.disk + "1") }}'
force: '{{ item.0.force|d(omit) }}'
fstype: '{{ item.0.fstype }}'
opts: '{{ item.0.fsopts|d(omit) }}'
with_together:
- '{{ disks_additional_disks }}'
- '{{ disks_stat.results }}'
when: item.1.stat.exists
tags: ['disks']
- name: "Disable periodic fsck and reserved space on ext3 or ext4 formatted disks"
environment:
PATH: "{{ ansible_env.PATH }}:/usr/sbin:/sbin"
shell: tune2fs -c0 -i0 -m0 {{ item.0.part | default(item.0.disk + "1") }}
with_together:
- '{{ disks_additional_disks }}'
- '{{ disks_stat.results }}'
when: "disks_additional_disks and ( item.0.fstype == 'ext4' or item.0.fstype == 'ext3' ) and item.0.disable_periodic_fsck|default(false)|bool and item.1.stat.exists"
tags: ['disks']
- name: "Ensure the mount directory exists"
file:
path: '{{ item.mount }}'
state: directory
with_items: '{{ disks_additional_disks }}'
tags: ['disks']
- name: "Get UUID for partition"
environment:
PATH: "{{ ansible_env.PATH }}:/usr/sbin:/sbin"
command: blkid -s UUID -o value {{ item.0.part | default(item.0.disk + "1") }}
check_mode: no
register: disks_blkid
with_together:
- '{{ disks_additional_disks }}'
- '{{ disks_stat.results }}'
changed_when: False
when: item.1.stat.exists
tags: ['disks']
- name: "Mount additional disks"
mount:
name: '{{ item.0.mount }}'
fstype: '{{ item.0.fstype }}'
opts: '{{ item.0.mount_options|d(omit) }}'
passno: '0'
src: 'UUID={{ item.1.stdout }}'
state: '{{ item.0.mount_state|d("mounted") }}'
with_together:
- '{{ disks_additional_disks }}'
- '{{ disks_blkid.results }}'
- '{{ disks_stat.results }}'
when: item.2.stat.exists
tags: ['disks']
register: disks_additional_disks_handler_notify
notify:
- restart services
- name: "Mount additional disks - nfs"
mount:
name: '{{ item.mount }}'
fstype: '{{ item.fstype }}'
opts: '{{ item.mount_options|d(omit) }}'
src: '{{ item.disk }}'
state: '{{ item.mount_state|d("mounted") }}'
when: item.fstype == 'nfs'
with_items: '{{ disks_additional_disks }}'
tags: ['disks']
register: disks_additional_disks_nfs_handler_notify
notify:
- restart services - nfs
- name: "Ensure the permissions are set correctly"
file:
path: '{{ item.mount }}'
owner: '{{ item.user | default("root") }}'
group: '{{ item.group | default("root") }}'
state: directory
with_items: '{{ disks_additional_disks }}'
when: item.user is defined or item.group is defined
tags: ['disk']
- meta: flush_handlers