logrotate
1 | /var/log/apache2/*.log { daily missingok rotate 366 compress compresscmd /bin/bzip2 compressext .bz2 notifempty create 640 root adm sharedscripts postrotate /etc/init.d/apache2 reload > /dev/null endscript } |
1 | /var/log/apache2/*.log { daily missingok rotate 366 compress compresscmd /bin/bzip2 compressext .bz2 notifempty create 640 root adm sharedscripts postrotate /etc/init.d/apache2 reload > /dev/null endscript } |
Curator est un outil qui va permettre de purger d’anciens indices ou snapshot ES
Need :
1 | ~# pip install elasticsearch-curator==2.1.2 |
1 | $> curator delete --prefix .marvel- --older-than 15 |
1 | 15 3 * * * elasticsearch /usr/local/bin/curator --host 127.0.0.1 delete --prefix logstash- --older-than 360 --time-unit days --timestring '%Y.%m.%d' |
1 | ~# curl -XGET 'http://localhost:9200/_cluster/health?pretty' |
1 | ~# curl -XGET 'localhost:9200/_nodes?pretty' |
1 | ~# curl -XGET 'localhost:9200/_stats?pretty' |
1 | # sed -i 's/false/true/' /etc/default/rsync # cat /etc/rsyncd.conf [opt] comment = repertoire opt path = /opt use chroot = yes lock file = /var/lock/rsyncd read only = yes list = yes uid = root gid = root strict modes = yes hosts allow = <IP DU VOISIN> ignore errors = no ignore nonreadable = no transfer logging = no log format = %t: host %h (%a) %o %f (%l bytes). Total %b bytes. timeout = 600 refuse options = checksum dry-run dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz |
Screen
L’escape Key : C-a
soit Control+a
Créer un screen :1
screen -S "un nom"
Créer un screen, a partir d’un .screenrc :1
screen -c .screenrc -S "un nom"
Lister les sessions :1
screen -ls
S’attacher à une session :1
screen -x "nom de session"
Quitter un screen :1
C-a d
Créer une nouvelle fenêtre :1
C-a c
Renommer une fenêtre :1
C-a a "nom"
Lister les fenêtres :1
C-a "
Fenêtre suivante :1
C-a n
Fenêtre précédente :1
C-a p
Killer une fenêtre :1
C-a k
Resizer la fenêtre :1
C-a F
=============
courses:
https://evantage.gilmoreglobal.com/?#/ (mypass)
Labs:
https://aws.qwiklab.com/live
https://aws.amazon.com/solutions/case-studies/
Regardez ce nouveau blog: https://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html. Ce nouveau “system manager” rend les choses encore plus faciles.
Il existe une variété de laboratoires autonomes ( self-paced labs (http://aws.amazon.com/training/self-paced-labs/)), qui acquièrent une expérience pratique sur de nombreux services AWS. Qu'il s'agisse d'un nouveau service ou d'un sujet dont vous avez entendu parler et que vous souhaitez mieux comprendre ou préparer un examen ou une discussion.
N'oubliez pas qu'AWS Free Tier vous permet de commencer à utiliser gratuitement AWS.
Vous pouvez également créer des alarmes de facturation pour contrôler la facturation sur votre utilisation de carte bleu.
Les pages Whitepapers(https://aws.amazon.com/whitepapers/) et Cloud Security(https://aws.amazon.com/security/) contiennent des informations détaillées sur la sécurité et d’autres sujets relatifs au cloud. Voici le lien vers AWS Well-Architected Framework(https://aws.amazon.com/architecture/well-architected/) et AWS Cloud Adoption Framework(https://aws.amazon.com/professional-services/CAF/) - deux très bonnes lectures!
Plus de formation, de laboratoires ou de certification ?
Acloud.guru
https://qwiklabs.com/
https://www.whizlabs.com
Learn to Build(https://aws.amazon.com/getting-started/use-cases/)
Besoin de plus d’informations sur les outils:
Simple Calculator(http://aws.amazon.com/calculator): Cette calculatrice vous aide à établir vos dépenses mensuelles.
Whitepapers(http://aws.amazon.com/whitepapers): Know more in-depth comparisons and guidelines from AWS Whitepapers
TCO Calculator(http://aws.amazon.com/tco-calculator/): Une calculatrice d’aperçu du coût total “On-prem” par rapport aux coût dans le cloud d’AWS
IAM policy simulator(https://policysim.aws.amazon.com/home/index.jsp): Vérifiez vos politiques avant d’intégrer votre code et votre IAM
DynamoDB plus d’info(https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/best-practices.html)
Voici des liens supplémentaires: AWS YouTube channel(https://www.youtube.com/user/amazonwebservices) , AWS Official Blog(http://aws.amazon.com/blogs/aws/), AWS SlideShare.net(http://www.slideshare.net/AmazonWebServices), et AWS Architecture Centre(http://aws.amazon.com/architecture/).
Regions + HAzone
and edge locations (dns deployement tools etc..)
T2 Credit accumulation
spare when not used so you can sometimes go over the limit
Spot Instances good offer ?
gather info on EC2
curl http://169.254.169.254/latest/meta-data
curl http://169.254.169.254/latest/user-data
== Stockage
Instance Store : Ephemeral, SSD
EBS : snapshot, persistance
SSD
SSD (provisioned iops)
Throughput Optimized 40Mb/s
S3 : can create endpoint on vpc subnet(where you manage policy)
configure notification (add, mod, delete)
possibility ti activate versionning
4 classes standard, standard infrequente access, glacier, (old : reduced redundancy)
glacier : use sdk to write ! or lifecycle rules
EFS : Share FS
== Network
DC to aws => AWS Direct Connect (bgp style ?)
Auto Assign pub IP != Elastic IP (fix)
IGW Internet GW
VGW Vpn GW
NGW Nat GW : EIP priv -> pub
VPC
VPC Peering : between vpcs (often use for peering an admin vpc) same region, not encrypt, no chaining route between vpc-A to vpc-C through vpc-B
Elastic LoadBalancer : RR, sticky or not, cross AZ (zone) or not (defaut not), TCP HTTP, autoscalable, can be internal lb
Route53 : dns w request w criteria : weight, healthcheck, latency, geo
== Security
Security groups : by instances , statefull, whitelist, Deny allways prior
NACL Network acl : by subnets, order list , blacklist, stateless
== Monitor
cloudwatch
gather metrics
can set alarm threshold
create filter
cloudtrail
log specific actions : tracing , also api
can chosse : Region or global
aws config
inventory with trigger alarm
config run every 15min and notice all change
redshift
big data stocakge en colonne
== Automation
AWS CloudFormation
Manage template
then -> stack
create AMI from instance (like snapshot)
tips : send message (w sqs) a the end of user-data
System Manager Service : run scripts etc…
Inspector : check for secu vulnerability
AutoScaling
auto healthcheck
auto add in ELB
auto balance in zones
can autoscale by steps
auscale lifecycle hooks (send hooks for start or stop)
can perform rolling update
Can autoscale 1
creating a scale group w 1 instance : will restart if healthcheck fail
aws-cfn-bootstrap : between ansible and cloudinit
Coudformer
create cloudFormation templates
parameter : ask human for value
mappings : dictio
== CLI
aws configuration
== LEXIC
EC2 = Elastic Compute Cloud
IAM = Identity Access Management
EBS = Elastic Block Store (Optimzed is dedicated)
S3 = Simple Storage Service
EIP = Elastic IP
AMI = Amazon Machine Image
ENI = Elastic Network Interfaces
SQS = Simple Queue Service
SNS = Simple Notification Service
ELB = Elastic Load Balancer
complete -C ‘/usr/local/bin/aws_completer’ aws
JMESPATH
query cli json
http://jmespath.org/
CostExplorer in billing section
Trusted adviser : some check for cost/secu
https://github.com/Netflix/aminator
EC2
ECS
Lambda (autoscale /!\ set max)
EBS - block
S3 - object
Glacier - slow
Snowball - transfert data (via box and ups)
Cloudfront - CDN
storage gateway - vm to install on-premise who transfert to aws S3 (eg : backup)
RDS - sql mysql psql oracle
Dynamo - nosql
DMS - Database Migration Service
Redshift - big data
VPC
Direct Connect - private link
Route 53 - DNS + LB
cloudwatch - monit
formation - deploy/template
trusted advisor - advise
A authentication
A authorization
A accouting
AIM
WAF
cloudtrail - log activity on managment
EMR - hadoop
kinesis - streaming
Api gateway - expose HTTP Lambda, dynamoDB, kinesis …
SQS - Simple Queue Service (1 to 1) max 256K (dead letter queue)
message delivery MINIMUM once !!
workspaces - desktop
Machine learning
iot - connect devices to cloud
Code pipeline - release automation
Code commit - control service, version
Code Build - build and test
Code deploy - deploy orchestration
X-ray - issued detector and optimization
Codestar - set and configure the developer tools
Cloud9 - web environment IDE
shield (standart) - DDOS protection
shield advanced + waf
M C T2 etc …
factu
Reserved Instances :
Standard = fixed
Convertible = modif instances type
Scheduled = not 100%
Intel techno
default 20/region
Type factu
factu par object
S3 gratuit en interne (az) (check if cloudfront more interesting)
Hook à l’upload/download
retention, deplacement (vers glacier) , modif facturation
versionning
S3 upload gratos
diff type
IA
RRS (old)
apprendre les 99,99..% for s3
encryption SSE-S3, SSE-KMS, SSE-C
SSE = server side
CSE = Client side
gerer le multi-part upload (chunk your file) ~ 100Mo
service regional
unique woldwide name
IA (minimum 30days)
can log all access
taille max
IO
Share storage (NFS 4.0/4.1)
Linux only
Types access :
expedited
standard
bulk
Vaults - archive (gros tar)
= retention
scalable (but not autoscalable)
Can use read replicas (but latency) - No available for oracle and M$sql
No root access(no mega tuning)
autoscale
no size limit
no throughtput limit
Lecture :
consistent
non consistent
Dynamo max name key/values combined => 400KB
Read 4KB/s
Write 1KB/s
Deny default
Delegate - tempo access to a role
important
ou (organisation unit) (ldap style)
endpoint in vpc for non-vpc-linked service
non transitif
no nat - think about lan ip range
nat priv => pub
en retour d’un LB
bound to range not instance
0.0.0.0/0 route
Capture traffic
-> cloudwatch (or S3)
customer gw. connect customer lan to vpc
= vpn
2 tunnels (ha)
switch to ec2
PUB or Private
http, tcp, websocket
TLS termination
ApplicationLB = HTTP(s) (manage web route)
NetworkLB = TCP
Drain = ability to graceful stop backend
logs -> cloudwatch
cloudwathc alarms -> evenement -> trigger
possibilité de fournir les logs cloudwatch à un outils de monitor tier
every 5min or every minutes
default metrics : cpu, disk read, network in
/!\ warmup (and cooldown)
deploiement not orchestration
mapping - dictionnaire | + parameters + conditions = un minimum d’intelligence et de “variablilité”
can use lambda to add functions
cloudformation groups - separate cloudformation script (ex : aim, vpc, appli, etc..)
deploy and scaling service for web appli
ready for production (or preproduction)
use puppet or chef
replace puppet or chef
aws or on premise
can Run Command
simple notification
(1 to multiple)
mail (plain / json)
SMS
HTTP(S)
SQS
Mobile push messaging
lambda
256K max
ActiveMQ fully managed
…
logs tcp connexions
set on : vpc, subnet, EIP
ex : “aws config” alarms if resources with no tag
inelligent threat detection
analyse : cloud trail , vpc flow, dns query logs
scan (from inside) : anti virus, best practice deviation, remote root login, vulnerable appli installed …
key master
encrypt keys with a master key, that i don’t own
Symmetric only
AI analyse data -> suggest data classification
Hardware Security Module
hard dedicated storage for your key
(kind of dedicated kms)
can create asymetric encryption
M$ ad
ad connector : connect to on premise ad
security token service
for temporary access to resources
sso using saml with amazon
Web Identify Federation
use a tier account to auth (ex, amazon, google, fb, etc..)
in another region :
backup/restore
pilot light = cold drp
low capacity stand by = drp low capacity (possiblity to lb 80%/20%)
active-active
1 | notify-send -u critical -t 5000 -a note -i /home/jerem/test/game.ico Message "hello world" |
1 | # tail /etc/crontab |
1 | # cat /root/sshkey_generate.sh |
1 | # /etc/init.d/rsyslogd stop |
1 | # for i in `cat /etc/hostname` ; do sed -i s/$i/vmtemplate/g /etc/hosts ; done |
== DRAFT ==
1 | # echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list |
On echange les clefs des users postgres entre les differents serveurs
1 | # tail -3 /etc/postgresql/9.5/main/pg_hba.conf |
1 | # diff /etc/postgresql/9.5/main/postgresql.conf /tmp/postgresql.conf.ori |
toute les commande repmgr sont a lancer en postgres
1 | # cat /etc/repmgr.conf |
1 | $ createuser -s repmgr |
1 | repmgr -f /etc/repmgr.conf primary register |
On check
1 | =# \c repmgr |
1 | repmgr -h node1 -U repmgr -d repmgr -D /var/lib/postgresql/9.5/main/ -f /etc/repmgr.conf standby clone |
On check (depuis le master)
1 | =# SELECT * FROM pg_stat_replication; |
1 | repmgr -f /etc/repmgr.conf standby register |
On check
1 | =# \c repmgr |
node1 est dead
1 | $ repmgr cluster show |
on stop ses services
1 | # /etc/init.d/postgresql stop |
on resync & register
1 | # repmgr -h node3 -U repmgr -d repmgr -D /var/lib/postgresql/9.5/main/ -f /etc/repmgr.conf standby clone |
node1 is back
1 | $ repmgr cluster show |
pour quand le script d’init marchera :
1 | # sed -i 's/REPMGRD_ENABLED=no/REPMGRD_ENABLED=yes/' /etc/default/repmgrd |
mon script d’init de remplacement :
1 | # cat /etc/init.d/repmgrd.init #!/bin/bash ### BEGIN INIT INFO # Provides: repmgrd # Required-Start: $remote_fs $all # Should-Start: # Required-Stop: # Default-Start: 2 3 5 # Default-Stop: 0 6 # Description: Starts and stops repmgrd. ### END INIT INFO REPMGRD=/usr/bin/repmgrd REPMGR_PIDFILE=/var/run/postgresql/repmgrd.pid REPMGRD_CONF=/etc/repmgr.conf case $1 in # Startup repmgrd servers. start) su postgres -c "$REPMGRD -m -d -p $REPMGR_PIDFILE -f $REPMGRD_CONF" #renice -5 -p `cat $REPMGR_PIDFILE` echo "starting..." ;; # Stop repmgrd servers. stop) if [ -f $REPMGR_PIDFILE ] then if ps auxww | grep `cat $REPMGR_PIDFILE` | grep -v grep then kill `cat $REPMGR_PIDFILE` echo "stopping..." else echo "Repmgrd is not running, but $REPMGR_PIDFILE is present" fi else echo "Repmgrd is not running" fi ;; # status status) if ps auxww | grep `cat $REPMGR_PIDFILE` | grep -v grep then echo "Repmgrd is running" else echo "Repmgrd is stopped" fi ;; *) echo "Init de Repmgrd: stop / start ou status" ;; esac |
repmgr.conf
%n - node ID
%e - event type
%s - success (1 or 0)
%t - timestamp
%d - details
event_notification_command=’/path/to/some/script %n %e %s “%t” “%d”‘
event_notifications=master_register,standby_register,witness_create
https://github.com/2ndQuadrant/repmgr/blob/master/README.md#generating-event-notifications-with-repmgrrepmgrd
Install1
# apt-get install heartbeat
conf de la vip1
# cat <<EOF >> /etc/network/interfaces
#auto ens160:pg # nonauto
iface ens160:pg inet static
address 10.0.69.0.120
netmask 255.255.255.0
EOF
# echo "net.ipv4.ip_nonlocal_bind=1" >> /etc/sysctl.conf
# sysctl -p
commande heartbeat utiles
/usr/share/heartbeat/hb_standby
/usr/share/heartbeat/hb_takeover
conf heartbeat
1 | # cat /etc/ha.d/ha.cf ############################### # logging logfile /var/log/ha/ha-log debugfile /var/log/ha/ha-debug ############################### # communication autojoin none udpport 694 ucast ens160 10.0.69.100 bcast ens160 ############################### # thresholds warntime 5 deadtime 15 initdead 60 keepalive 2 ############################### # nodes node bdd1 node bdd2 auto_failback off |
1 | # cat /etc/ha.d/haresources # VIP bdd1 IPaddr::10.0.69.120/24/ens160:pg #MailTo::moi@pouet.fr::HA_Postgresql |
on adapte la conf /etc/repmgr.conf
1 | # diff /etc/repmgr.conf_heartbeat /etc/repmgr.conf_ori - promote_command='repmgr standby promote -f /etc/repmgr.conf' - follow_command='repmgr standby follow -f /etc/repmgr.conf' + promote_command='sudo /usr/share/heartbeat/hb_takeover && repmgr standby promote -f /etc/repmgr.conf' + follow_command='sudo /usr/share/heartbeat/hb_standby && repmgr standby follow -f /etc/repmgr.conf' |
conf sudo
postgres ALL=(root) NOPASSWD:/usr/share/heartbeat/hb_takeover
Generer le hash du pass1
# grub-mkpasswd-pbkdf2
editer la conf1
# cat /etc/grub.d/40_custom
#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries. Simply type the
# menu entries you want to add after this comment. Be careful not to change
# the 'exec tail' line above.
# define superusers
set superusers="toor"
#define users
password_pbkdf2 toor grub.pbkdf2.sha512.10000.41A4BB99DD5F334FDZFOFDVDPVA33F6FABB7AEEE9EE3F6B533D47B05712619797411812BD97413274F9F29FAD1A444D3FA5E4B0619F8425C316120BC248480F29A1BDE64619.E61C7C6A45BF4A0E89143868AC6B2874E2DC0BDCF0BC7764B5C77ECD293E8C8743E0A
authoriser a booter sans pass1
# diff /etc/grub.d/10_linux /etc/grub.d/10_linux.ori
130c130
< echo "menuentry '$(echo "$title" | grub_quote)' ${CLASS} \$menuentry_id_option 'gnulinux-$version-$type-$boot_device_id' --unrestricted {" | sed "s/^/$submenu_indentation/"
---
> echo "menuentry '$(echo "$title" | grub_quote)' ${CLASS} \$menuentry_id_option 'gnulinux-$version-$type-$boot_device_id' {" | sed "s/^/$submenu_indentation/"
< echo "menuentry '$(echo "$os" | grub_quote)' ${CLASS} \$menuentry_id_option 'gnulinux-simple-$boot_device_id' --unrestricted {" | sed "s/^/$submenu_indentation/"
---
> echo "menuentry '$(echo "$os" | grub_quote)' ${CLASS} \$menuentry_id_option 'gnulinux-simple-$boot_device_id' {" | sed "s/^/$submenu_indentation/"
On regenere la conf1
# update-grub
1 | fdisk -l # pour vous aider à trouver la partition sur laquelle est votre ubuntu mount /dev/sdaX /mnt # montage de celle-ci en remplaçant le X par le bon numéro de partition mount --bind /dev /mnt/dev # lien symbolique du dossier /dev en cours d'utilisation vers le disque monté mount --bind /dev/pts /mnt/dev/pts # lien symbolique du dossier /dev/pts en cours d'utilisation vers le disque monté mount --bind /sys /mnt/sys # lien symbolique du dossier /sys en cours d'utilisation vers le disque monté mount -t proc /proc /mnt/proc # Pour que Grub2 trouve /proc/mounts chroot /mnt /bin/bash # mise à la racine du disque monté mount -a # montage des partitions dans le chroot apt-get install grub-pc # installation du logiciel Grub2 (sur le disque maintenant à la racine) update-grub # création d'un nouveau fichier de configuration : grub.cfg grub-install /dev/sda # installation de grub sur le MBR |