# coding: utf-8

# # Deep eXtreme MultiLabel (DXML) Classification
#
# A classification task is defined as 'extreme' when the number of classes considered grows considerably (from a dozen to millions of classes). The 'Multi-label' problem is similar to a classification task, but each sample may have more than one associated label. Furthermore, any sample may have a different number of associated labels.
#
# This notebook aims to prove that naive approaches may not be neither performant nor mathematically correct for this problem, and to show a simplified version of the XML method developed in paper [**Deep Extreme Multi-label Learning**](https://arxiv.org/pdf/1704.03718.pdf) by Zhang et al.

# In[1]:

# Imports
get_ipython().magic('matplotlib inline')
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
from sklearn.naive_bayes import GaussianNB
from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestRegressor

# # Theory: label independance
#
# A multi-label problem is different from a multi-class one as the latter only attributes the most probable class for a sample, whereas the multi-label problem needs to attribute the most probable subset of classes.
#
# For instance, in visual recognition, a multi-class problem would be to recognise the main subject of a picture whereas a multi-label problem would be the recognition of all recognizable objects in the picture.
#
# At first one could think that the subset of the most probable classes would form the most probable subset of classes, but it is not true on the general case. It can only be true if each class probability is independent from the others (or $P(A\,|\, B) = P(A) $ for each A,B classes in our problem). **This hypothesis can't possibly be satisfied in most of our problems**.
#
# That's what makes mult-label so tricky. Indeed the number of possible subsets is much larger than the number of classes (growing as $2^N$, with N classes. That means multilabelling with 100 labels makes a little more than a 1 million classes classification problem. Scaling to 100 labels would have us have to choose between $ 1,26 . 10^{30}$ possible subsets.
#
# Thus there's a need for a reasonable approach.

# # Birds chant recognition
#
# We'll follow through this reasoning using data from [The 9th annual MLSP competition: New methods for acoustic classification of multiple simultaneous bird species in a noisy environment]( http://vintage.winklerbros.net/Publications/mlsp2013ma.pdf ).
#
# This data represents audio recordings of birds chants, with the objective of recognising up to 19 different birds. It has been split for the sake of this exercise.

# In[19]:

# Get data
X_train, X_test = np.loadtxt('samples/X_train_birds.txt'), np.loadtxt('samples/X_test_birds.txt')
y_train, y_test = np.loadtxt('samples/y_train_birds.txt'), np.loadtxt('samples/y_test_birds.txt')
birds_list = ["Brown Creeper","Pacific Wren","Pacific-slope Flycatcher","Red-breasted Nuthatch",
              "Dark-eyed Junco","Olive-sided Flycatcher","Hermit Thrush","Chestnut-backed Chickadee",
              "Varied Thrush","Hermit Warbler","Swainson's Thrush","Hammond's Flycatcher",
              "Western Tanager","Black-headed Grosbeak","Golden Crowned Kinglet","Warbling Vireo",
              "MacGillivray's Warbler","Stellar's Jay","Common Nighthawk"]
print(X_train.shape)

#
Let's look at the data to picture the multi-label problem. Observe and comment on the data: check that this is a multi-label problem. What problems could arise if we applied a global "blind" method?

# In[3]:

print(len(X_train), ' train samples and ', len(X_test), ' test sample')
print('###')
for i in range(len(birds_list)):
    print(' Species {} appears {} times'.format(birds_list[i], np.sum(y_train[:,i]) + np.sum(y_test[:,i])))
print('###')
all_y = np.concatenate([y_train, y_test])
all_sum_y = np.sum(all_y, axis=1)
print('As a mean {} species appear on each recording (standard deviation {})'.format(np.mean(all_sum_y), np.std(all_sum_y)))

# Remember that the repartion of occurence between species is quite unequal, which would tend to bias black-block approaches

#
(Optional) Implement a Multi-Layer Perceptron (MLP) classifier to check the difficulty of the task.

#
# ## Mathematically false approach: parallel classifiers
#
#
The intuitive approach that supposes independance of different labels would be to implement parallel classifiers. Let's look at their performance.

# In[4]:

# CORRECTION 1 : SKLEARN
mlpc = MLPClassifier(hidden_layer_sizes=(64,32,32), activation='relu')
mlpc.fit(X_train, y_train)
mlpc.score(X_test, y_test), mlpc.score(X_train, y_train)

# In[5]:

# CORRECTION 2 : KERAS

from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.optimizers import Adam

mlpk = Sequential()

#model.add(BatchNormalization())
#model.add(Activation('relu'))

mlpk.add(Dense(32, activation='relu', input_shape=(X_train.shape[1],)))
mlpk.add(Dense(32, activation='relu'))
mlpk.add(Dense(len(birds_list), activation='tanh'))

mlpk.compile(loss='categorical_crossentropy', metrics=['categorical_accuracy'], optimizer=Adam())

h = mlpk.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=20, verbose=0)

# In[6]:

# CORRECTION
plt.figure()
plt.plot(h.history['categorical_accuracy'], c='blue', label='train')
plt.plot(h.history['val_categorical_accuracy'], c='red', label='test')
plt.legend(loc='best')
plt.title('Accuracy of the MLP')
plt.show()

# # Definitely, MLPs are not very convincing!

# ## Baseline: parallel classifiers
#
#
The intuitive approach (which you now know is inexact!): Individually predict each label (for example, use Naive Bayes classifiers).

# In[8]:

# CORRECTION
classif_list = []
train_score_list, test_score_list = [], []

# Training
for i in range(len(birds_list)):
    y_train_for_this_bird = y_train[:,i]
    new_classifier = GaussianNB()
    new_classifier.fit(X_train, y_train_for_this_bird)
    classif_list.append(new_classifier)

# Test & display results
for i in range(len(birds_list)):
    classif = classif_list[i]
    train_score = float('{0:.3f}'.format(classif.score(X_train, y_train[:,i])))
    test_score = float('{0:.3f}'.format(classif.score(X_test, y_test[:,i])))
    train_score_list.append(train_score)
    test_score_list.append(test_score)
    print('Detecting {} with {}% accuracy (training {}%)'.format(birds_list[i], 100*test_score, 100*train_score))

predict_train = np.zeros_like(y_train)
predict_test = np.zeros_like(y_test)
for i in range(len(birds_list)):
    classif = classif_list[i]
    predict_train[:,i] = classif.predict(X_train)
    predict_test[:,i] = classif.predict(X_test)
acc_train = 1 - np.sum(np.abs(predict_train - y_train))/(y_train.shape[0]*y_train.shape[1])
acc_test = 1 - np.sum(np.abs(predict_test - y_test))/(y_test.shape[0]*y_test.shape[1])
print('###')
print('Global accuracy: testing {}, training {}'.format(acc_test, acc_train))

well_labeled = 0
for i in range(len(y_train)):
    if np.sum(np.abs(y_train[i,:] - predict_train[i,:])) == 0:
        well_labeled +=1
print('Overall {} out of the {} training samples were well labeled'.format(well_labeled,len(y_train)))

well_labeled = 0
for i in range(len(y_test)):
    if np.sum(np.abs(y_test[i,:] - predict_test[i,:])) == 0:
        well_labeled +=1
print('Overall {} out of the {} testing samples were well labeled'.format(well_labeled,len(y_test)))

# Of course, in addition to being mathematically frowned upon, this method is cumbersome,
# imagine having to build a million classifiers! It is not at all extensible to extreme classification

#
Remember that those results derive from an inadequate method. The Multi - Label classification and its XL version, the eXtreme MultiLabel Classification (XML) are difficult to treat problems on which the classical methods can not be used as they are.

# ## Embedding 101
#
# Embedding is a projection of data in another space. As we can specifically design this destination space, we'll be able to highlight specific relation between data points that may have been hidden before.
#
# For instance:
#
#

#

#
#         Dog
#         Cat
#         Elephant
#         Turtle
#         Fox
#

#

#         Dog
#         1
#         0
#         0
#         0
#         0
#

#

#         Cat
#         0
#         1
#         0
#         0
#         0
#

#

#         Fox
#         0
#         0
#         0
#         0
#         1
#

#

#
# could be projected as:
#
#

#

#
#         Dimension 1
#         Dimension 2
#         Dimension 3
#         Dimension 4
#

#

#         Dog
#         0.123
#         0.452
#         0.865
#         0.852
#

#

#         Cat
#         0.458
#         0.741
#         0.852
#         0.963
#

#

#         Fox
#         0.789
#         0.456
#         0.845
#         0.654
#

#

#
# Embedding is used a lot in Natural Language Processing to create mathematically just syntaxical or lexical relations, but in this problem, it will enable us to relate one labels to another.

# # Zhang et. al. method : Interaction between labels Embedding
#
# Their method can be decomposed in 4 very simple steps, as we'll dispense from the 'deep' part thanks to our relatively small problem.
#
# ### First step: compute labels adjacency
#
# Let's look at our labels to see how close they are to each other. We'll draw the adjacency matrix and adjacency graph. In the latter, each node is a label and a vertex symbolizes the number of conjoint appearances of two labels.
#
# ### Second step: learning adjacency-based label embedding
#
# The cited paper uses DeepWalk to form an embedding based on the adjacency graph.
#
# In our example here we'll do with a linear projection using the adjacency matrix, which despite being much more simple will prove its value.
#
# ### Third step : prediction in embedded space
#
# Instead of doing a classification problem from the features to the labels, we'll train a multidimensional regression agent from the features to the embedded space, which will yield coordinates for a data sample in this continuous space.
#
# ### Fourth step : return to labels
#
# From our embedded space coordinates, we'll run a k-Nearest Neighbors (kNN) to find the closest labels and return them as a prediction.
#
#
# Here's the original, full-scale full-deep method as proposed in the original article:
#

# ## On our example
#
# ### Compute adjacency matrix
#
#
#
#
 Let's build the labels proximity matrix as $M_{[i,j] \in [|1,19|]^2}$ so that $m_{i,j}$ is the joint apparition frequency of labels $i$ and $j$.

# In[20]:

# Trick
#Visualizing a 19 * 19 matrix is not necessarily obvious.
#You will find below a simple method to represent a matrix as an image.

my_matrix = np.random.normal(loc=1, scale=0.3, size=(25,25)) # matrix to represent
plt.figure()
plt.imshow(my_matrix,cmap="hot")
plt.colorbar()
plt.title('Visualization of the matrix')
plt.show()

# In[10]:

#We'll rather visualize an image than a 19*19 matrix.

matrix = np.zeros((y_train.shape[1],y_train.shape[1]))
edges =[]
for row in y_train :
    for i in range(len(row)):
        for j in range(len(row)):

            if row[i]*row[j] == 1: # both labels are activated
                matrix[i,j] += 1
                if ([i,j] not in edges) and ([j,i] not in edges) and not i==j:
                    edges.append([i,j])

#to get frequency we'll divide by number of appearances
norm_matrix = np.copy(matrix)
# np.save(norm_matrix,'norm_matrix.npm')
for i in range(matrix.shape[0]):
    d = norm_matrix[i,i]
    norm_matrix[:,i] = norm_matrix[:,i]/d
    norm_matrix[i,:] = norm_matrix[i,:]/d
    norm_matrix[i,i] = 1

# Visualize
plt.figure()
plt.imshow(norm_matrix,cmap="hot")
plt.colorbar()
plt.title('Labels proximity (absolute)')
plt.show()

# Diagonal coefficients are not interesting, let's ditch them
norm_matrix -= np.identity(norm_matrix.shape[0])
plt.figure()
plt.imshow(norm_matrix,cmap="hot")
plt.colorbar()
plt.title('Label proximity (without diagonal)')
plt.show()

# ### Adjacency graph
#
# This graph is actually used in Zhang et al's method, but for us it will only be a data visualization tool.
#
# Here's the **draw_graph** function that could be used in other projects too.
#
# Another way of representing the proximity between labels is to construct a weighted graph. This is necessary for Zhang et al, who use this graph for their embedding, but ours will only require the adjacency matrix. We recommend that you build this graph at a minimum in order to visualize the data.
#
# We put at your disposal the function draw_graph which will automatically build it from a list of connected labels and the proximity matrix you just calculated. The threshold option allows you to choose from how many occurrences a link will be represented on the graph.

# In[11]:

import networkx as nx #For those among you that use Anaconda dist, you already got this

def draw_graph(edges, weights_matrix=None, threshold=0, figsize=(8,8)):

    '''
    edges : lists of connected labels
    weights_matrix : labels proximity matrix
    threshold : number of occurence required to draw a connection
    figsize : size of the plot to display
    '''

    edges = [edge for edge in edges if weights_matrix[edge[0],edge[1]] > threshold]

    # additional settings (you can mess around here)
    node_size = 1600
    node_color = 'blue'
    node_alpha = 0.2
    node_text_size = 12
    edge_color = 'blue'
    edge_alpha= 0.3
    edge_tickness = 1
    edge_text_pos = 0.3
    text_font = 'sans-serif'

    plt.figure(figsize=figsize)
    # create networkx graph
    G=nx.Graph()

    # add edges
    for edge in edges:
        G.add_edge(edge[0], edge[1])

    # select shell autolocation
    graph_pos=nx.shell_layout(G)

    # draw graph
    nx.draw_networkx_nodes(G,
                           graph_pos,
                           node_size=node_size,
                           alpha=node_alpha,
                           node_color=node_color)
    nx.draw_networkx_edges(G, graph_pos,width=edge_tickness, alpha=edge_alpha,edge_color=edge_color)
    nx.draw_networkx_labels(G, graph_pos,font_size=node_text_size, font_family=text_font)

    # construct weights dict
    weights={}
    for i in range(len(edges)):
        weights[tuple(edges[i])] = weights_matrix.astype(int)[edges[i][0],edges[i][1]]

    # draw weights
    edge_labels = weights
    nx.draw_networkx_edge_labels(G, graph_pos, edge_labels=edge_labels,
                                 label_pos=edge_text_pos)

    # show graph
    plt.title('Proximity graph between labels with a proximity threshold at {} occurrences'.format(threshold))
    plt.show()

# Exemple
graph = [(0, 1), (1, 4), (2, 3), (0, 4), (1, 3)]
weights_matrix = np.random.randint(1,high=5,size=(5,5))
draw_graph(graph,weights_matrix)
draw_graph(graph,weights_matrix, threshold=2)

#
 Let's look at some graphs

# In[12]:

# CORRECTION

draw_graph(edges, matrix, figsize=(13,13), threshold=0)
draw_graph(edges, matrix, figsize=(8,8), threshold=5)
draw_graph(edges, matrix, figsize=(8,8), threshold=9)

# ## Design the embedding
#
# Zhang et al use [**Deepwalk**](https://github.com/phanein/deepwalk) on the adjacency graph to build their embeding.
#
# Once the graph is constructed, Zhang et al use Deepwalk to reduce the size of their labels, that is, project them into a continuous space of smaller size.
#
# At our scale, we can content ourselves with making a simple multiplication of our labels vectors by the standardized proximity matrix that we have constructed previously, which will project our labels in a continuous space of dimension 19. It is in this space Continuing that we are going to do a regression learning.
#
# In our problem, multiplying the label vectors by the normalized adjacency matrix will project our labels into a continuous space of 19 dimensions. Our regression will be conducted in this space.
#
# For the moment, we are going to set up the Embedding in this continuous space. Construct a ** embedding_encode ** function that multiplies one or more samples by the matrix to the continuous space (adjacency matrix) and then centers and collapses; apply it to y_train and y_test.
#
# For example, we want to create a function such that: f ([0,0,0,0,0,0,1,0,0,1]) = [0.25, 0.32, 0.87]

# In[13]:

def embedding_encode(y, matrix, scalor=None, return_scalor=False):
    y_conti = np.dot(y, norm_matrix)
    if scalor:
        y_conti=scalor.transform(y_conti)
    else:
        scalor = StandardScaler()
        y_conti = scalor.fit_transform(y_conti)

    if return_scalor:
        return y_conti, scalor
    else:
        return y_conti

y_conti_train, scalor = embedding_encode(y_train, norm_matrix, return_scalor=True)
y_conti_test = embedding_encode(y_test, norm_matrix, scalor=scalor)

# The decoding part will be performed by a kNN research in the continuous space.
#
#
 The * KNearestNeighborsClassifier * class you can import from * sklearn.neighbors * has a * k neighbors * method that returns the nearest n $ $ indices. Use it to create a ** encoding_decode ** method that lets you move from a sample of continuous space to the label space.

#
# (If you prefer to implement directly or otherwise the KNN algorithm, it's to your taste!)
#
# For example, we want to create a function such that: f ([0.25, 0.32, 0.87]) = [0,0,0,0,0,0,1,1,0,0,1]

# In[14]:

knn = KNeighborsClassifier()
knn.fit(y_conti_train, y_train)

def embedding_decode(y, knn, neighbors_number=2, algorithm='brute', y_train=y_train):
    indexes_of_neighbors = knn.kneighbors(y,n_neighbors=neighbors_number, return_distance=False)
    number_of_samples = indexes_of_neighbors.shape[0]
    labels = np.zeros((number_of_samples, y_train.shape[1]))

    for i in range(number_of_samples):
        for j in range(neighbors_number):
            labels[i, :] += y_train[indexes_of_neighbors[i,j],:]
        for j in range(labels.shape[1]):
            if labels[i,j]: labels[i,j] = 1

    return labels

# small optimization on neighbors_number
err_list=[]
err_list_t=[]
for n in range(2,30):
    y_train_decoded = embedding_decode(y_conti_train, knn, n )
    y_test_decoded = embedding_decode(y_conti_test, knn , n)
    error = np.sum(np.abs(y_train_decoded - y_train))
    error_t = np.sum(np.abs(y_test_decoded - y_test))
    err_list.append(error/(y_train.shape[0]*y_train.shape[1]))
    err_list_t.append(error_t/(y_test.shape[0]*y_test.shape[1]))
plt.figure()
plt.plot(err_list, label='training data')
plt.plot(err_list_t, label='testing data', color='red')
plt.legend(loc='best')
plt.title('Error from number of selected neighbours')
plt.show()

# ## Multidimensional Regression
#
# We now have a way to move from the space of the labels to their continuous representation and vice versa, it remains for us to implement the regression in the continuous space to complete the method!
#
#
 Create a regressor (for example, use RandomForestRegressor) predicting continuous values (* y_conti *) from features (* X_train, X_test *), and then use the ** embedding_decode ** function previously coded to return to the label space. Compare the accuracy obtained with that of your parallel classifiers.

#
# We'll implement a simple model to predict coordinates in our embedded space from features, such as a random forest.

# In[15]:

get_ipython().run_cell_magic('time', '', '# mini-optimization\n# This estimator is the one who realizes the true "prediction", a good precision is more than desirable\n# We thus realize a mini-optimization on the number of arbs of our Random Forest\nlist_train, list_test = [],[]\nfor n in range(100,1101,200):\n    regressor = RandomForestRegressor(n_estimators=n, n_jobs=-1)\n    regressor.fit(X_train, y_conti_train)\n    list_train.append(regressor.score(X_train,y_conti_train))\n    list_test.append(regressor.score(X_test,y_conti_test))\nplt.figure()\nplt.plot(list_train, label=\'score training\')\nplt.plot(list_test, label=\'score testing\', color=\'red\')\nplt.legend(loc=\'best\')\nplt.show()')

# In[16]:

# actual regression
regressor = RandomForestRegressor(n_estimators = 400, n_jobs=-1)
regressor.fit(X_train, y_conti_train)
y_conti_predict = regressor.predict(X_test)

# In[17]:

# Conversion in labels
y_predict = embedding_decode(y_conti_predict, knn)

# In[18]:

# Evaluate the full method precision
well_labeled = 0
for i in range(len(y_test)):
    if np.array_equal(y_test[i,:] , y_predict[i,:]):
        well_labeled +=1
print('Overall {} out of the {} testing samples were well labeled'.format(well_labeled,len(y_test)))

#
Even though our embedding is dramatically simple and the precision of our regressor is more than questionable, taking the labels proximity into account took our global precision from 24 to 146 well labeled samples out of 323. That's a way to show mathemaical correctness is also a practical help!

#
#
 Although our embedding is reduced to its simplest expression and our regressor is a questionable prediction, taking into account the adjacency of the labels has made it possible to pass from 24 to 141 well-ranked samples on 323 compared to our baseline of 19 parallel naive-bayes classifiers. The added value of the method is undeniable both mathematically and practically!

# The main limitation of the method is related to the function passing from the embedding space to the space of the labels. Indeed, the k-Nearest-Neighbors function takes as attribute the number of neighbors to be selected. This number is therefore fixed for all the samples. This implies that the same number of labels will be predicted for all samples.
#
# # For further details:
#
# Here are more XML datasets to have fun with:
#     - http://mulan.sourceforge.net/datasets-mlc.html
#     - http://manikvarma.org/downloads/XC/XMLRepository.html
#
# You can try on these to build the adjacency graph and use [**Deepwalk**](https://github.com/phanein/deepwalk) on it !
#
# Deepwalk is an algorithm that uses random walks to learn a representation of the nodes of a graph. Deepwalk uses the information obtained after random walks by considering the sequence of nodes encountered. This method has an advantage, it is adaptable. It is not necessary to restart the process from the beginning to add new nodes and relationships.


#!/usr/bin/env python
# coding: utf-8

# In[2]:

import os
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader
#from src.utils import get_max_lengths, get_evaluation
from han_in_dataset_bert import HanInDataset
from han_hierarchical_attn_bert import HierAttn

from tensorboardX import SummaryWriter
from nltk import word_tokenize, sent_tokenize
from sklearn import metrics
from sklearn.metrics import classification_report, roc_auc_score
import argparse
from tqdm.notebook import tqdm
import csv
import shutil
import numpy as np
import pandas as pd

# In[3]:

from text_with_topic_words_w_3_context_w_demog import handleTask, handleMergingCaseControlDataset

# In[4]:

def get_args():
    # for amazon_review_polarity dataset
    parser= argparse.ArgumentParser()
    parser.add_argument("--batch_size", type= int, default=16)
    parser.add_argument("--num_epochs", type= int, default= 100)
    parser.add_argument("--lr", type=float, default=0.001)
    parser.add_argument("--momentum", type=float, default=0.9)
    parser.add_argument("--word_hidden_size", type=int, default= 128)
    parser.add_argument("--sent_hidden_size", type=int, default= 128)
    parser.add_argument("--es_min_delta", type=float, default=0.0,
                        help="Early stopping's parameter: minimum change loss to qualify as an improvement")
    parser.add_argument("--es_patience", type=int, default=35,
                        help="Early stopping's parameter: number of epochs with no improvement after which training will be stopped. Set to 0 to disable this technique.")
    parser.add_argument("--train_set", type=str, default="./../../dataset/data/amazon_review_polarity_csv/train.csv")
    parser.add_argument("--test_set", type=str, default="./../../dataset/data/amazon_review_polarity_csv/test.csv")
    parser.add_argument("--test_interval", type=int, default=1, help="Number of epoches between testing phases")
    parser.add_argument("--word2vec_path", type=str, default="./../../dataset/data/glove.6B.50d.txt")
    parser.add_argument("--log_path", type=str, default="tensorboard/han_voc")
    parser.add_argument("--saved_path", type=str, default="./../../checkpoints/saved_models")

    args, unknown = parser.parse_known_args()

    return args

# In[5]:

def get_max_lengths(data_path):
    word_length_list = []
    sent_length_list = []
    with open(data_path) as csv_file:
        reader = csv.reader(csv_file, quotechar='"')
        for idx, line in tqdm(enumerate(reader)):
            text = ""
            for tx in line[1:]:
                text += tx.lower()
                text += " "
            sent_list = sent_tokenize(text)
            sent_length_list.append(len(sent_list))

            for sent in sent_list:
                word_list = word_tokenize(sent)
                word_length_list.append(len(word_list))

        sorted_word_length = sorted(word_length_list)
        sorted_sent_length = sorted(sent_length_list)

    return sorted_word_length[int(0.8*len(sorted_word_length))], sorted_sent_length[int(0.8*len(sorted_sent_length))]

# In[6]:

def get_evaluation(y_true, y_prob, list_metrics):
    y_pred = np.argmax(y_prob, -1)
    output = {}
    if 'accuracy' in list_metrics:
        output['accuracy'] = metrics.accuracy_score(y_true, y_pred)
    if 'loss' in list_metrics:
        try:
            output['loss'] = metrics.log_loss(y_true, y_prob)
        except ValueError:
            output['loss'] = -1
    if 'confusion_matrix' in list_metrics:
        output['confusion_matrix'] = str(metrics.confusion_matrix(y_true, y_pred))

    try:
        print(classification_report(y_true, y_pred))
        print("AUC= ",roc_auc_score(y_true, y_pred, average="macro", multi_class="ovo", labels=[0,1,2]))
    except:
        print('An exception occurred while generating classification report')

    return output

# In[16]:

import nltk
def handleTopicSent(dataset: pd.DataFrame, topic_set: pd.DataFrame):
    all_topic_words= topic_set[["0","1","2"]].values.tolist()
    all_topic_words=[word for subset in all_topic_words for word in subset ]
    ori_len=[]
    short_len=[]
    for i,doc in tqdm(enumerate(dataset["text"])):
        temp_doc=[]
        #print("--------->",doc)
        doc=nltk.sent_tokenize(doc)
        ori_len.append(len(doc))
        for each_sent in doc:
            found=[word for word in nltk.word_tokenize(each_sent) if word in set(all_topic_words)]
            if len(found) !=0:
                #print(found)
                #print(each_sent)
                temp_doc.append(each_sent)
        short_len.append(len(temp_doc))
        #print("==========>", " ".join(temp_doc))
        dataset.loc[i, "text"]=(" ".join(temp_doc))
    ori_avg_len=sum(ori_len)/len(ori_len)
    short_avg_len=sum(short_len)/len(short_len)
    print(ori_avg_len)
    print(short_avg_len)
    return dataset

# In[17]:

def handleDataSupport(all_dataset:pd.DataFrame,topic_set:pd.DataFrame,with_context:bool,with_demog:bool, all:bool, window_len:int, cluster_num:int):
    #with context
    full_dataset=handleTopicSent(all_dataset, topic_set) # imported from text_with_topic_words.ipynb file
    full_dataset["text"].replace(to_replace=[r"-"," +"], value=[" "," "],inplace=True, regex=True)
    #pure_dataset=full_dataset.dropna().reset_index(drop=True) # dropping null value
    #full_dataset=all_dataset
    # full_dataset.rename(columns={"TEXT_REPROT":"text"}, inplace=True)
    # transformed_label_list=[1 if i== cluster_num else 0 for i in full_dataset["case"].tolist()]
    # full_dataset["label"]=transformed_label_list
    #full_dataset.head(5)

    print(full_dataset.shape)
    full_dataset.to_csv("./temp_full_dataset.csv")
    full_dataset=pd.read_csv("./temp_full_dataset.csv")
    full_dataset=full_dataset.iloc[:,1:]
    full_dataset=full_dataset.dropna().reset_index(drop=True)
    print(full_dataset.shape)

    df0=[]
    df1=[]
    df2=[]
    for i in tqdm(range(len(full_dataset))):
        #print(int(full_dataset.at[i,"GAF"]))
        if int(full_dataset.at[i,"label"])==0:
            #print("hu")
            df0.append(full_dataset.iloc[i,:])
        elif int(full_dataset.at[i,"label"])==1:
            df1.append(full_dataset.iloc[i,:])
            #print("ku")
        else:
            df2.append(full_dataset.iloc[i,:])
    df0=pd.concat(df0, axis=1, ignore_index=True).T
    df1=pd.concat(df1, axis=1, ignore_index=True).T
    df2=pd.concat(df2, axis=1, ignore_index=True).T
    print(df0.shape, df1.shape, df2.shape)

    eq_val=2000

    df_train=pd.concat([df0.iloc[:eq_val,:],df1.iloc[:eq_val,:],df2.iloc[:eq_val,:]], ignore_index=True)
    df_valid=pd.concat([df0.iloc[eq_val:,:],df1.iloc[eq_val:,:],df2.iloc[eq_val:,:]], ignore_index=True)

    df_train = df_train.sample(frac=1, random_state=42)
    df_valid = df_valid.sample(frac=1, random_state=42)

    df_train.to_csv("./temp_42_tr3-.csv")
    df_train=pd.read_csv("./temp_42_tr3-.csv")
    df_train=df_train.iloc[:,1:]
    df_train=df_train.dropna().reset_index(drop=True)
    print(df_train.shape)

    df_valid.to_csv("./temp_42_val3-.csv")
    df_valid=pd.read_csv("./temp_42_val3-.csv")
    df_valid=df_valid.iloc[:,1:]
    df_valid=df_valid.dropna().reset_index(drop=True)
    print(df_valid.shape)

    #df_test=df_test.reset_index(drop=True)

    #print(df_test.head(5))
    return df_train, df_valid

# In[9]:

def train(opt, train_set, valid_set, embd_size, model_name):
    if torch.cuda.is_available():
        torch.cuda.manual_seed(123)
    else:
        torch.manual_seed(123)

    output_file= open(opt.saved_path+ os.sep + "logs.txt", "w")
    output_file.write("Model Parameters: {}".format(vars(opt)))
    training_params={"batch_size":opt.batch_size, "shuffle": True, "drop_last": True}
    test_params={"batch_size":opt.batch_size, "shuffle": False, "drop_last": True}
    #max_word_length, max_sent_length= get_max_lengths(opt.train_set)
    max_word_length=30
    max_sent_length=60
    num_classes= 3
    training_set= HanInDataset(train_set, opt.word2vec_path, max_sent_length, max_word_length, model_name) # tuple(ndarray(encoded_doc), label)
    print(training_set[0])
    training_generator= DataLoader(training_set, **training_params)

    test_set = HanInDataset(valid_set, opt.word2vec_path, max_sent_length, max_word_length, model_name)
    test_generator = DataLoader(test_set, **test_params)

    # defining the model
    model = HierAttn(opt.word_hidden_size, opt.sent_hidden_size, opt.batch_size, num_classes,
                       opt.word2vec_path, max_sent_length, max_word_length, embd_size, model_name)

    if os.path.isdir(opt.log_path):
        shutil.rmtree(opt.log_path)
    os.makedirs(opt.log_path)
    writer = SummaryWriter(opt.log_path)
    # writer.add_graph(model, torch.zeros(opt.batch_size, max_sent_length, max_word_length))

    if torch.cuda.is_available():
        model.cuda()

    criterion = nn.CrossEntropyLoss() # can handle both binary and multiclass classification
    optimizer = torch.optim.SGD(filter(lambda p: p.requires_grad, model.parameters()), lr=opt.lr, momentum=opt.momentum)
    best_loss = 1e5
    best_epoch = 0
    model.train()   # model.train() tells your model that we are training the model.
                    # This helps inform layers such as Dropout and BatchNorm, which are designed to behave differently during training and evaluation.
                    # For instance, in training mode, BatchNorm updates a moving average on each new batch;
                    # whereas, for evaluation mode, these updates are frozen.
    num_iter_per_epoch = len(training_generator)
    for epoch in tqdm(range(opt.num_epochs)):
        for iter, (feature, label) in enumerate(training_generator):
            if torch.cuda.is_available():
                feature = feature.cuda()
                label = label.cuda()
            optimizer.zero_grad()
            model._init_hidden_state()
            # here, feature is numpy array
            predictions = model(feature)    # predicting from the model

            loss = criterion(predictions, label)
            loss.backward()
            optimizer.step()
            training_metrics = get_evaluation(label.cpu().numpy(), predictions.cpu().detach().numpy(), list_metrics=["accuracy"])
            print("Epoch: {}/{}, Iteration: {}/{}, Lr: {}, Loss: {}, Accuracy: {}".format(
                epoch + 1,
                opt.num_epochs,
                iter + 1,
                num_iter_per_epoch,
                optimizer.param_groups[0]['lr'],
                loss, training_metrics["accuracy"]))
            writer.add_scalar('Train/Loss', loss, epoch * num_iter_per_epoch + iter)
            writer.add_scalar('Train/Accuracy', training_metrics["accuracy"], epoch * num_iter_per_epoch + iter)
        if epoch % opt.test_interval == 0:
            model.eval()
            loss_ls = []
            te_label_ls = []
            te_pred_ls = []
            for te_feature, te_label in test_generator:
                num_sample = len(te_label)
                if torch.cuda.is_available():
                    te_feature = te_feature.cuda()
                    te_label = te_label.cuda()
                with torch.no_grad():
                    model._init_hidden_state(num_sample)
                    te_predictions = model(te_feature)
                te_loss = criterion(te_predictions, te_label)
                loss_ls.append(te_loss * num_sample)
                te_label_ls.extend(te_label.clone().cpu())
                #te_predictions = F.softmax(te_predictions)
                te_pred_ls.append(te_predictions.clone().cpu())
                #print(te_predictions)

            te_loss = sum(loss_ls) / test_set.__len__()
            te_pred = torch.cat(te_pred_ls, 0)
            te_label = np.array(te_label_ls)
            test_metrics = get_evaluation(te_label, te_pred.numpy(), list_metrics=["accuracy", "confusion_matrix"])

            output_file.write(
                "Epoch: {}/{} \nTest loss: {} Test accuracy: {} \nTest confusion matrix: \n{}\n\n".format(
                    epoch + 1, opt.num_epochs,
                    te_loss,
                    test_metrics["accuracy"],
                    test_metrics["confusion_matrix"]))
            print("Epoch: {}/{}, Lr: {}, Loss: {}, Accuracy: {}".format(
                epoch + 1,
                opt.num_epochs,
                optimizer.param_groups[0]['lr'],
                te_loss, test_metrics["accuracy"]))
            writer.add_scalar('Test/Loss', te_loss, epoch)
            writer.add_scalar('Test/Accuracy', test_metrics["accuracy"], epoch)
            model.train()
            if te_loss + opt.es_min_delta < best_loss:
                best_loss = te_loss
                best_epoch = epoch
                torch.save(model, opt.saved_path + os.sep + "whole_model_han")

            # Early stopping
            if epoch - best_epoch > opt.es_patience > 0:
                print("Stop training at epoch {}. The lowest loss achieved is {}".format(epoch, te_loss))
                break

# In[19]:

if __name__== "__main__":
    # with joined atc mapping (10,25)
    with_context=True
    with_demog=False
    all=True
    window_len=2
    model_name= "prajjwal1/bert-mini"
    embd_size= 256
    opt = get_args()

    all_dataset= pd.read_csv("./../../../generated_files/bipolar_ori_partitioned_atc_mapped_data.csv")

    topic_set= pd.read_csv("./../../../generated_files/bipolar_topic_words.csv")

    #print(topic_set)

    all_dataset= all_dataset.iloc[:,1:]

    all_dataset.rename(columns={"TEXT_REPROT": "text", "GAF": "label"}, inplace=True)
    all_datasett=all_dataset.iloc[:, lambda df: [4, 5]]
    all_datasett= all_datasett.sample(frac=1, random_state=42).iloc[:,:].reset_index(drop=True)
    df_train, df_valid=handleDataSupport(all_datasett,topic_set.iloc[:25,:], with_context,with_demog, all, window_len, 0)

    train(opt, df_train,df_valid,embd_size, model_name)

# In[ ]:

# In[11]:

df_train.head()

# In[12]:

df_train.at[0,"text"]

# In[13]:

#------------------------ P L A Y G R O U N D -------------------------#
# a=torch.Tensor([[1,2,3],[4,5,6],[7,8,9],[10,11,12]])
# b=torch.tensor([[11,22,33],[44,55,66],[77,88,99],[100,200,300]])

# aa=a.unsqueeze(1)
# print(aa.shape)
# bb=b.unsqueeze(1)
# c=torch.cat((aa,bb),1)
# print(c)
# print(c.size())

# d,e=torch.split(c,[1,1],1)
# print(d.squeeze(1).shape)


# Merge Two Binary Trees

# Definition for a binary tree node.
class TreeNode:
    def __init__(self, x):
        self.val = x
        self.left = None
        self.right = None

    def printval(self):
        if self.left:
            self.left.printval()
        print(self.val)
        if self.right:
            self.right.printval()

class Solution:
    def mergeTrees(self, t1: TreeNode, t2: TreeNode) -> TreeNode:
        if not t1 and t2:
            return t2
        elif not t2 and t1:
            return t1
        elif not t2 and not t1:
            return None
        else:
            ans = TreeNode(t1.val + t2.val)
            ans.left = self.mergeTrees(t1.left, t2.left)
            ans.right = self.mergeTrees(t1.right, t2.right)
            return ans

if __name__ == '__main__':
    t1 = TreeNode(1)
    t1.left = TreeNode(3)
    t1.right = TreeNode(2)
    t1.left.left = TreeNode(5)
    t2 = TreeNode(2)
    t2.left = TreeNode(1)
    t2.right = TreeNode(3)
    t2.left.right = TreeNode(4)
    t2.right.right = TreeNode(7)
    s = Solution()
    s.mergeTrees(t1,t2).printval()


from selenium import webdriver
from bs4 import BeautifulSoup
import time
import re

class LimitNotMatched(Exception):
    def __init__(self, error_msg):
        super().__init__(error_msg)

class GooglePlayStoreReviewCrawler:
    """
    Google play에서 리뷰를 크롤링하는 class
    """
    def __init__(self, app_name:str, chrome_driver_url:str, url:str, conn, limit_scroll:bool = False, limit_num:int= -1, scroll_pause:int= 2):
        """

        :param app_name: 크롤링 할 어플리케이션 이름
        :param chrome_driver_url: 크롬 드라이버의 위치
        :param url: 크롤링 할 페이지 url
        :param conn: DB 연결
        :param limit_scroll: 스크롤 제한 설정
        :param limit_num: 스크롤 제한 수
        :param scroll_pause: 스크롤 동작 시 잠시 멈추는 시간

        """
        self.app_name = app_name
        self.__chrome_driver_url = chrome_driver_url
        self.url = url
        self.limit_scroll = limit_scroll
        self.limit_num = limit_num
        self.conn = conn
        self.scroll_pause = scroll_pause
        self.__make_table()

    def __make_table(self):
        """
        Table이 존재하지 않으면 생성
        :return: None
        """
        cur = self.conn.cursor()
        sql = '''
        CREATE TABLE IF NOT EXISTS {} (
            user_id VARCHAR(50),
            total_score TINYINT,
            review_score TINYINT,
            write_date DATE,
            useful_cnt SMALLINT,
            review_txt LONGTEXT
        );
        '''.format(self.app_name)
        cur.execute(sql)
        self.conn.commit()
        cur.close()

    def start(self):
        """
        크롤링을 진행하는 함수
        :return: None
        """
        if self.limit_scroll is False and self.limit_num != -1:
            raise LimitNotMatched("limit_scroll is False, but you enter limit_num.")

        cur = self.conn.cursor()

        driver = webdriver.Chrome(self.__chrome_driver_url)
        driver.get(self.url)

        last_height = driver.execute_script("return document.body.scrollHeight")
        scroll_cnt = 0

        while True:
            if self.limit_scroll is True and scroll_cnt >= self.limit_num:
                break

            driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
            time.sleep(self.scroll_pause)
            new_height = driver.execute_script("return document.body.scrollHeight")

            if new_height == last_height:
                try:
                    driver.find_element_by_xpath("//*[contains(@class,'U26fgb O0WRkf oG5Srb C0oVfc n9lfJ')]").click()
                except:
                    break

            last_height = new_height
            scroll_cnt += 1

        page = driver.page_source

        bs = BeautifulSoup(page, "lxml")
        rs = bs.find_all('div',{'class':'d15Mdf bAhLNe'})

        for rs_i in rs:
            name = rs_i.find('div', {'class': 'bAhLNe kx8XBd'}).find('span', {'class': "X43Kjb"}).text
            score = rs_i.find('div', {'class': 'pf5lIe'}).find('div').get('aria-label')
            total_score, review_score = re.findall('\d', score)
            date = rs_i.find('span', {'class': "p2TkOb"}).text
            date = '/'.join(re.findall('[0-9]+', date))
            useful_cnt = rs_i.find('div', {'class': 'YCMBp GVFJbb'}).find('div', {'class': "XlMhZe"}).find('div', {
                'class': "jUL89d y92BAb"}).text
            review_txt = rs_i.find('div', {'class': 'UD7Dzf'}).find('span', {'jsname': 'fbQN7e'}).text

            if len(review_txt) == 0:
                review_txt = rs_i.find('div', {'class': 'UD7Dzf'}).find('span', {'jsname': 'bN97Pc'}).text

            review_txt = review_txt.replace("'","\\'")

            sql = 'INSERT INTO {} VALUES (\'{}\',{},{},\'{}\',{},\'{}\')'.format(self.app_name, name, total_score,
                                                                     review_score, date, useful_cnt, review_txt)
            try:
                cur.execute(sql)
            except Exception as e:
                print('Error Occured in {} {} \n:{}'.format(name,date,e))
                print(sql)
        self.conn.commit()
        driver.close()
        print('Finished')


from typing import Final
from telegram import Update
from telegram.ext import CommandHandler, MessageHandler, filters, ContextTypes, Updater, Application

TOKEN: Final = "token"
BOT_USERNAME: Final = "username"

# Commands

async def start_command(update: Update, context: ContextTypes.DEFAULT_TYPE):
    await update.message.reply_text('Hello!')

async def help_command(update: Update, context: ContextTypes.DEFAULT_TYPE):
    await update.message.reply_text('Please type something so I can respond!')

async def set_message_command(update: Update, context: ContextTypes.DEFAULT_TYPE):
    await update.message.reply_text("Please enter the message you want to set:")
    context.user_data["waiting_for_message"] = True

async def send_announcement_command(update: Update, context: ContextTypes.DEFAULT_TYPE):
    custom_message = context.user_data.get("custom_message")
    if custom_message:
        group_ids = [5693780887, -1967973417]  # Replace with your group chat IDs
        for group_id in group_ids:
            await context.bot.send_message(group_id, custom_message)
        await update.message.reply_text("Announcement sent to groups!")
    else:
        await update.message.reply_text("No custom message set. Use /setmessage first.")

# Responses

async def handle_new_message(update: Update, context: ContextTypes.DEFAULT_TYPE):
    if context.user_data.get("waiting_for_message"):
        message = update.message.text
        context.user_data["waiting_for_message"] = False
        context.user_data["custom_message"] = message
        await update.message.reply_text("Message set successfully!")

async def handle_message(update: Update, context: ContextTypes.DEFAULT_TYPE):
    message_type: str = update.message.chat.type
    text: str = update.message.text

    print(f'User ({update.message.chat.id}) in {message_type}: "{text}"')

def main():
    print("Starting bot...")

    app = Application.builder().token(TOKEN).build()

    # Register command handlers
    app.add_handler(CommandHandler("start", start_command))
    app.add_handler(CommandHandler("help", help_command))
    app.add_handler(CommandHandler("setmessage", set_message_command))
    app.add_handler(CommandHandler("announce", send_announcement_command))

    # Register message handler
    app.add_handler(MessageHandler(filters.TEXT, handle_new_message))
    app.add_handler(MessageHandler(filters.TEXT, handle_message))  # Add this line to process regular messages

    print("Polling...")
    app.run_polling(poll_interval=3)

if __name__ == "__main__":
    main()


# Cgroup tune script
# Example:
# python tuneCgroups.py --cgName "usr_xpus-sched2.scope" --cgController "memory" --memory.limit_in_bytes 40466547916 --memory.memsw.limit_in_bytes 49056482508 --memory.soft_limit_in_bytes 36419893125

import os
import os.path
import shutil
import psutil
import json
import time
import argparse
import sys

from xcalar.external.LegacyApi.XcalarApi import XcalarApi
from xcalar.external.LegacyApi.WorkItem import WorkItem
from xcalar.compute.util.config import build_config
from xcalar.compute.coretypes.Status.ttypes import StatusT
from xcalar.external.LegacyApi.Env import XcalarConfigPath
from xcalar.external.client import Client
from xcalar.external.exceptions import XDPException
from xcalar.compute.services.Cgroup_xcrpc import Cgroup, CgRequest
from xcalar.external.LegacyApi.XcalarApi import XcalarApiStatusException

# Args parsing
parser = argparse.ArgumentParser()
parser.add_argument("--cgName", type=str, help="Cgroup name", dest='cgname')
parser.add_argument("--cgController", type=str, help="Cgroup controller name", dest='controller')
parser.add_argument("--memory.limit_in_bytes", type=int, help="Cgroup controller memory.limit_in_bytes", dest='limit_in_bytes')
parser.add_argument("--memory.memsw.limit_in_bytes", type=int, help="Cgroup controller memory.memsw.limit_in_bytes", dest='swlimit_in_bytes')
parser.add_argument("--memory.soft_limit_in_bytes", type=int, help="Cgroup controller memory.soft_limit_in_bytes", dest='soft_limit_in_bytes')
args = parser.parse_args()

def listCgroupParams():
    client = Client(bypass_proxy=True)
    cgserv = Cgroup(client)
    inputObj = {}
    inputObj["func"] = "listCgroupsConfigs"
    cgreq = CgRequest()
    cgreq.jsonInput = json.dumps(inputObj)
    cgres = cgserv.process(cgreq)
    return json.loads(cgres.jsonOutput)

def getCgroupParams(cgName, cgController):
    client = Client(bypass_proxy=True)
    cgserv = Cgroup(client)
    inputObj = {}
    inputObj["func"] = "getCgroup"
    inputObj["cgroupName"] = cgName
    inputObj["cgroupController"] = cgController
    cgreq = CgRequest()
    cgreq.jsonInput = json.dumps(inputObj)
    cgres = cgserv.process(cgreq)
    cgParamDict = json.loads(json.loads(
        cgres.jsonOutput)[0][0])[cgController][0]
    for k, v in cgParamDict.items():
        cgParamDict[k] = str(v).rstrip()
    print("\n\nGet cgName: {}, cgController: {}, Params {}".format(cgName, cgController, cgParamDict))
    return cgParamDict

def setCgroupParams(cgName, cgController, cgParams):
    client = Client(bypass_proxy=True)
    cgserv = Cgroup(client)
    inputObj = {}
    inputObj["func"] = "setCgroup"
    inputObj["cgroupName"] = cgName
    inputObj["cgroupController"] = cgController
    inputObj["cgroupParams"] = cgParams
    cgreq = CgRequest()
    cgreq.jsonInput = json.dumps(inputObj)
    cgres = cgserv.process(cgreq)
    print("\n\nSet cgName: {}, cgController: {}, Params {}".format(cgName, cgController, cgParams))

# List Cgroup Params
listOut = listCgroupParams()
print("***\n\t\tList Cgroup Params: {}".format(listOut))

# Get all XPU Cgroup params
print("***\n\t\tGet XPUs current config")
xpuMemSysCgSched0 = getCgroupParams("sys_xpus-sched0.scope", "memory")
xpuCpuSysCgSched0 = getCgroupParams("sys_xpus-sched0.scope", "cpu")
#xpuCpusetSysCgSched0 = getCgroupParams("sys_xpus-sched0.scope", "cpuset")
xpuCpuacctSysCgSched0 = getCgroupParams("sys_xpus-sched0.scope", "cpuacct")

xpuMemSysCgSched1 = getCgroupParams("sys_xpus-sched1.scope", "memory")
xpuCpuSysCgSched1 = getCgroupParams("sys_xpus-sched1.scope", "cpu")
#xpuCpusetSysCgSched1 = getCgroupParams("sys_xpus-sched1.scope", "cpuset")
xpuCpuacctSysCgSched1 = getCgroupParams("sys_xpus-sched1.scope", "cpuacct")

xpuMemSysCgSched2 = getCgroupParams("sys_xpus-sched2.scope", "memory")
xpuCpuSysCgSched2 = getCgroupParams("sys_xpus-sched2.scope", "cpu")
#xpuCpusetSysCgSched2 = getCgroupParams("sys_xpus-sched2.scope", "cpuset")
xpuCpuacctSysCgSched2 = getCgroupParams("sys_xpus-sched2.scope", "cpuacct")

xpuMemUsrCgSched0 = getCgroupParams("usr_xpus-sched0.scope", "memory")
xpuCpuUsrCgSched0 = getCgroupParams("usr_xpus-sched0.scope", "cpu")
#xpuCpusetUsrCgSched0 = getCgroupParams("usr_xpus-sched0.scope", "cpuset")
xpuCpuacctUsrCgSched0 = getCgroupParams("usr_xpus-sched0.scope", "cpuacct")

xpuMemUsrCgSched1 = getCgroupParams("usr_xpus-sched1.scope", "memory")
xpuCpuUsrCgSched1 = getCgroupParams("usr_xpus-sched1.scope", "cpu")
#xpuCpusetUsrCgSched1 = getCgroupParams("usr_xpus-sched1.scope", "cpuset")
xpuCpuacctUsrCgSched1 = getCgroupParams("usr_xpus-sched1.scope", "cpuacct")

xpuMemUsrCgSched2 = getCgroupParams("usr_xpus-sched2.scope", "memory")
xpuCpuUsrCgSched2 = getCgroupParams("usr_xpus-sched2.scope", "cpu")
#xpuCpusetUsrCgSched2 = getCgroupParams("usr_xpus-sched2.scope", "cpuset")
xpuCpuacctUsrCgSched2 = getCgroupParams("usr_xpus-sched2.scope", "cpuacct")

# Set given config
print("\n\t\t***Setting Config")
if ("xpus" not in args.cgname or args.controller != "memory" or args.limit_in_bytes == 0 or args.swlimit_in_bytes == 0 or args.soft_limit_in_bytes == 0):
    print("Invalid input params")
    sys.exit(-1)
curParams = getCgroupParams(args.cgname, args.controller)
newCgParams = {}
newCgParams["memory.limit_in_bytes"] = args.limit_in_bytes
newCgParams["memory.memsw.limit_in_bytes"] = args.swlimit_in_bytes
newCgParams["memory.soft_limit_in_bytes"] = args.soft_limit_in_bytes
setCgroupParams(args.cgname, args.controller, newCgParams)
print("\n\t\t***Getting Config")
getCgroupParams(args.cgname, args.controller)


import test
import asposetaskscloud
from test.base_test_context import BaseTestContext

class TestApiCoverage(BaseTestContext):

    def test_api_coverage(self):
        arr = [test.TestAssignments, test.TestCalendarExceptions, test.TestCalendarWorkWeeks, test.TestCalendars,
               test.TestDocumentProperties, test.TestExtendedAttributes, test.TestFile, test.TestFolder,
               test.TestGetPageCount, test.TestImportProject, test.TestOutlineCodes, test.TestProjectOnline,
               test.TestReadProject, test.TestRecalculation, test.TestRecurringInfo, test.TestReport,
               test.TestResources, test.TestStorage, test.TestTaskDocumentFormat, test.TestTaskLinks, test.TestTasks,
               test.TestTasksExtendedAttributes, test.TestTasksPrimaveraProperties, test.TestTimephasedData,
               test.TestVba, test.TestWbs]
        test_methods = []
        uncovered_methods = ''
        for ar in arr:
            test_methods += list(filter(lambda x: not (x.startswith('__') and x.endswith('__')) and x != 'test_folder',
                                        ar.__dict__.keys()))
        api_methods = list(filter(lambda x: not (x.startswith('__') and x.endswith('__'))
                                            and not x.endswith('with_http_info')
                                            and not x.endswith('letter')
                                            and not x.endswith('token'),
                                  asposetaskscloud.TasksApi.__dict__.keys()))
        for api_method in api_methods:
            if not 'test_' + api_method in test_methods:
                uncovered_methods += api_method + ' '
        self.assertTrue(len(uncovered_methods) == 0,
                        "There are methods you have to cover with tests: " + ''.join(uncovered_methods))


# %% [markdown]
# # Part One & Two

# %%
file = open('./Input/Day8_Input.txt', 'r')

contents = file.read()
data = contents.split("\n")
scenic_scores = []

# %%
def get_edge_tree_ct(row_ct):
    edge_tree_ct = 0
    ct = 0

    while ct < row_ct:
        if (ct == 1) or (ct == row_ct - 1): # First or last row of trees
            edge_tree_ct = edge_tree_ct + len(data[ct])
        else:
            edge_tree_ct = edge_tree_ct + 2

        ct = ct + 1

    return edge_tree_ct

def new_front_back_arr(row_ct, row_ix):
    front_back_row_ixs = []
    ct = 0

    while ct < row_ct:
        if ct != row_ix:
            front_back_row_ixs.append(ct)

        ct = ct + 1

    return front_back_row_ixs

def new_left_right_arr(tree_ct, tree_ix):
    left_right_tree_ixs = []
    ct = 0

    while ct < tree_ct:
        if ct != tree_ix:
            left_right_tree_ixs.append(ct)

        ct = ct + 1

    return left_right_tree_ixs

def get_viewing_distance(tree_ix, row_ix, tree_ct_per_row, row_ct, direction, tree):
    viewing_distance = 0

    if direction == 'left' or direction == 'front':
        increment_ix = tree_ix - 1 if direction == 'left' else row_ix - 1

        if increment_ix >= 0:
            while increment_ix >= 0:
                comp_tree = int(data[row_ix][increment_ix]) if direction == 'left' else int(data[increment_ix][tree_ix])
                viewing_distance = viewing_distance + 1

                if comp_tree >= tree:
                    break

                increment_ix = increment_ix - 1
    else:
        increment_ix = tree_ix + 1 if direction == 'right' else row_ix + 1
        end_ix = tree_ct_per_row - 1 if direction == 'right' else row_ct - 1

        if increment_ix <= end_ix:
            while increment_ix <= end_ix:
                comp_tree = int(data[row_ix][increment_ix]) if direction == 'right' else int(data[increment_ix][tree_ix])
                viewing_distance = viewing_distance + 1

                if comp_tree >= tree:
                    break

                increment_ix = increment_ix + 1

    return viewing_distance

def get_scenic_score(row_ix, tree_ix, tree_ct_per_row, row_ct, tree):
    left_viewing_distance = get_viewing_distance(tree_ix, row_ix, tree_ct_per_row, row_ct, 'left', tree)
    right_viewing_distance = get_viewing_distance(tree_ix, row_ix, tree_ct_per_row, row_ct, 'right', tree)
    front_viewing_distance = get_viewing_distance(tree_ix, row_ix, tree_ct_per_row, row_ct, 'front', tree)
    back_viewing_distance = get_viewing_distance(tree_ix, row_ix, tree_ct_per_row, row_ct, 'back', tree)

    scenic_score = left_viewing_distance * right_viewing_distance * front_viewing_distance * back_viewing_distance

    return scenic_score

def analyze_trees(data):
    row_ct = len(data)
    tree_ct_per_row = len(data[0])
    trees_w_visibility = 0
    row_ix = 1

    edge_tree_ct = get_edge_tree_ct(row_ct)

    while row_ix <= row_ct - 2: # Second-second to last row of trees
        row = data[row_ix]

        # Get all row indexes other than the one we're currently on
        front_back_row_ixs = new_front_back_arr(row_ct, row_ix)

        tree_ix = 1

        while tree_ix <= tree_ct_per_row - 2: # Second-second to last tree within the row
            tree = int(data[row_ix][tree_ix])
            left_right_tree_ixs = new_left_right_arr(tree_ct_per_row, tree_ix)

            # Check left-right tree visibility
            left_visibility = True
            right_visibility = True

            for lr in left_right_tree_ixs:
                compare_ix = int(lr)
                compare_tree = int(data[row_ix][lr])

                if compare_ix < tree_ix:
                    # Tree is to the left
                    if compare_tree >= tree:
                        left_visibility = False
                else:
                    # Tree is to the right
                    if compare_tree >= tree:
                        right_visibility = False

            # Check front-back tree visibility
            front_visibility = True
            back_visibility = True

            for fb in front_back_row_ixs:
                compare_ix = int(fb)
                compare_tree = int(data[fb][tree_ix])

                if compare_ix < row_ix:
                    # Tree is in front
                    if compare_tree >= tree:
                        front_visibility = False
                else:
                    # Tree is behind
                    if compare_tree >= tree:
                        back_visibility = False

            # Add current tree if visible from either the front, back, left, or right
            if ((left_visibility == True) or (right_visibility == True)
                or (front_visibility == True) or (back_visibility == True)):
                trees_w_visibility = trees_w_visibility + 1

            # Get the scenic score for this tree (part two)
            scenic_score = get_scenic_score(row_ix, tree_ix, tree_ct_per_row, row_ct, tree)
            scenic_scores.append(scenic_score)

            tree_ix = tree_ix + 1

        row_ix = row_ix + 1

    visible_trees = trees_w_visibility + edge_tree_ct

    return visible_trees

# %%
visible_trees = analyze_trees(data)

print(f"Answer (part one) - How many trees are visible from outside of the forest: {visible_trees}")
print(f"Answer (part two) - Highest scenic score possible: {max(scenic_scores)}")


import math

N = int(input())
positions = []
for _ in range(N):
    s = input()
    x = float(s.split(" ")[0])
    y = float(s.split(" ")[1])

    # we optimise the function f below by putting these in polar co-ordinates, so we only have to do 1 trig operation each time
    r = math.sqrt(x*x + y*y)
    if x == 0:
      theta = math.pi / 2
    else:
      theta = math.atan2(y, x)
    positions.append((r, theta))

def f(theta):
    values = []
    for (r, theta2) in positions:
        values.append(r * math.cos(theta - theta2))
    return max(values) - min(values)

MAX_ERROR = 0.0000005
INTERVALS = 200
# the period of the function is pi, but we'll take two intervals in case the lowest point is near an endpoint
(lower, upper) = (0, 2*math.pi)

diff = 1

while diff > MAX_ERROR / 2:
    # improve the interval
    length = (upper - lower) / INTERVALS
    inputs = [lower + i*length for i in range(INTERVALS + 1)]
    points_on_graph = [(x, f(x)) for x in inputs]

    outputs = [x[1] for x in points_on_graph]

    min_output = min(outputs)
    max_output = max(outputs)
    diff = max_output - min_output
    print("%.12f %.12f" % (diff, length))
    if diff <= MAX_ERROR / 2:
        break

    # find the input which gives the lowest output
    lowest_point = points_on_graph[0]
    for point in points_on_graph[1:]:
        if point[1] < lowest_point[1]:
            lowest_point = point

    (lower, upper) = (lowest_point[0] - length, lowest_point[0] + length)

print("%6f" % min_output)


# Bigrams language model baseline sentence generator (ML-2)
# Marcov's algorithm for producing sentences from a starting word based on probabilities

# Abderrahim BOUHRIZ
# Rania BOUAITA

import numpy as np
import sys
np.set_printoptions(threshold=sys.maxsize)

# open the corpus
def openfile():
    file = open("testfile.txt", "r")
    corpus = file.read().split("\n")
    corpus = [corpus]
    return corpus

# get bigrams from the corpus without occurrences
def getBigrams(corpus):
    Bigrams = []
    for bigram in corpus[0]:
        bigram = bigram.split("\t")
        Bigrams = np.append(Bigrams,bigram[0])
    return Bigrams

# get successors from bigrams without occurrences
def getSuccessors(Bigrammes):
    successors=[]
    for bigram in Bigrammes:
        unigram = bigram.split(" ")
        successors = np.append(successors,unigram[0]) # get the first word
        successors = np.append(successors,unigram[1]) # get the second word
    successors = list(dict.fromkeys(successors)) # remove repetitive words
    return successors

# get the number of occurrences of a given bigram
def getNumberOfOccurence(bigram):
    bigramOccurence = makeDictFromCorpus()
    if bigram not in bigramOccurence.keys():
        return 0
    return bigramOccurence[bigram]

# make a dictionary from a corpus
def makeDictFromCorpus():
    bigramOccurence ={}
    corpus=openfile()
    for i in corpus[0]:
        elements = i.split("\t")
        bigramOccurence[elements[0]] = elements[1]
    return bigramOccurence

# get unigrams from bigrams with the occurrences of each bigram
def getUnigramsFromBigrams():
    unigramsOccurence = {}
    bigramsOccurence = makeDictFromCorpus()
    for k, v in bigramsOccurence.items():
        elements = k.split(" ")
        if elements[0] in unigramsOccurence.keys():
            unigramsOccurence[elements[0]] += int(v)
        else:
            unigramsOccurence[elements[0]] = int(v)

        if elements[1] in unigramsOccurence.keys():
            unigramsOccurence[elements[1]] += int(v)
        else:
            unigramsOccurence[elements[1]] = int(v)
    return unigramsOccurence

# create a matrix based on the probabilities of the Markov chain
# A Markov chain is a model that describes a sequence of possible events in which
# the probability of each event depends only on the state reached in the previous event.
def makeMatrix(Bigrams,successors):
    size_x = len(Bigrams) # the bigrams are rows of the matrix
    size_y = len(successors) # the successors are columns of the matrix
    matrix = np.zeros((size_x,size_y))
    for i in range(size_x):
        for j in range(size_y):
            mot_big = Bigrams[i].split(" ")
            condidat= mot_big[1]+" "+successors[j] # concatenate the second word of the bigram with its successor
            occurenceOfBigram=getNumberOfOccurence(condidat) # count the number of occurrences of word(t) after word(t-1)
            if(int(occurenceOfBigram) == 0): # avoid division on zero
                matrix[i][j] = 0
            else:
                occurenceOfUnigram=getUnigramsFromBigrams()[mot_big[1]]
                #calculate the probability (the number of occurrences of bigram on the number of occurrences of successor)
                matrix[i][j] = int(occurenceOfBigram) / int(occurenceOfUnigram)
    return matrix

# Get the bigram index of the matrix from the given word
def getBigramIndexFromMatrix(word):
    indices = []
    bigramsOccurence = makeDictFromCorpus()
    unigramsOccurence = []
    for k in bigramsOccurence.keys():
        elements = k.split(" ")
        unigramsOccurence = np.append(unigramsOccurence,elements[1])
    for i in range(len(unigramsOccurence)):
        if(word ==unigramsOccurence[i]):
            indices.append(i)
    return indices

# Get the successor index of the given word based on the row of the matrix that has the maximum value.
def getSuccessorIndexFromSuccessors(matrix,i,successors):
    max= 0
    for j in range(len(successors)):
        if(matrix[i][j]>=max):
            max = matrix[i][j]
            indexJ = j
    return indexJ

# from the indices i and j, we predict the next word of a given word
def makeSentence(successors,matrix,word,size):
    sentence = word;
    for k in range(int(size)-1):
            i = getBigramIndexFromMatrix(word)
            j = getSuccessorIndexFromSuccessors(matrix,i[0],successors)
            sentence += " " + successors[j]
            word = successors[j]
    return sentence

# menu where we call the function that generates the sentence
def menu(successors,matrix):
    while True:
        word = input("Enter the start word: ");
        size = input("Choose the size of the sentence (5, 10, 15): ")
        sentence = makeSentence(successors,matrix,word,size)
        print(sentence)

# main program
def main():
    corpus = openfile()
    bigrams = getBigrams(corpus)
    successors = getSuccessors(bigrams)
    matrix = makeMatrix(bigrams, successors)
    menu(successors,matrix)
main()


class Solution:
    def longestConsecutive(self, nums: List[int]) -> int:
        if not nums: return 0
        lo = min(nums)
        nums = list(map(lambda x: str(x-lo).zfill(10), set(nums)))
        for i in range(10):
            buckets = []
            for _ in range(10):
                buckets.append([])
            for j in nums:
                buckets[int(j[-i-1])].append(j)
            nums.clear()
            for b in buckets:
                nums.extend(b)
        nums = list(map(int, nums))
        curr_el, curr_cnt = -1, 0
        best = 0
        for i in nums:
            if curr_el + 1 == i:
                curr_cnt += 1
            else:
                best = max(best, curr_cnt)
                curr_cnt = 1
            curr_el = i
        return max(best, curr_cnt)


# PyPoll
# In this challenge, you are tasked with helping a small, rural town modernize its vote-counting process. (Up until now, Uncle Cleetus had been trustfully tallying them one-by-one, but unfortunately, his concentration isn't what it used to be.)
# You will be give a set of poll data called [election_data.csv](PyPoll/Resources/election_data.csv). The dataset is composed of three columns: `Voter ID`, `County`, and `Candidate`. Your task is to create a Python script that analyzes the votes and calculates each of the following:

# The total number of votes cast
# A complete list of candidates who received votes
# The percentage of votes each candidate won
# The total number of votes each candidate won
# The winner of the election based on popular vote.

# As an example, your analysis should look similar to the one below:
#   ```text
#   Election Results
#   -------------------------
#   Total Votes: 3521001
#   -------------------------
#   Khan: 63.000% (2218231)
#   Correy: 20.000% (704200)
#   Li: 14.000% (492940)
#   O'Tooley: 3.000% (105630)
#   -------------------------
#   Winner: Khan
#   -------------------------
#   ```
# In addition, your final script should both print the analysis to the terminal and export a text file with the results.

import pandas as pd

df = pd.read_csv("PyPoll.csv", header = 'infer')
# print(df.columns)

print("Election Results")
print("-------------------------")
# Calculate the total number of votes
total_votes = df['Voter ID'].count()
print(f"Total Votes: {total_votes}")
print("-------------------------")

# print the percent of voters to the candidate khan
khan_candidate_as_true = df['Candidate'] == 'Khan'
# Create a df with only Khan
khan_df = df[khan_candidate_as_true]
# Count the number of votes Khan received
khan_votes = khan_df['Voter ID'].count()
khan_percent_votes = round(khan_votes / total_votes * 100.0, 3)
print(f"Khan: {khan_percent_votes}% ({khan_votes})")

# print the percent of voters to the candidate Correy
Correy_candidate_as_true = df['Candidate'] == 'Correy'
# Create a df with only Correy
Correy_df = df[Correy_candidate_as_true]
# Count the number of votes Correy received
Correy_votes = Correy_df['Voter ID'].count()
Correy_percent_votes = round(Correy_votes / total_votes * 100.0, 3)
print(f"Correy: {Correy_percent_votes}% ({Correy_votes})")

# print the percent of voters to the candidate Li
Li_candidate_as_true = df['Candidate'] == 'Li'
# Create a df with only Li
Li_df = df[Li_candidate_as_true]
# Count the number of votes Li received
Li_votes = Li_df['Voter ID'].count()
Li_percent_votes = round(Li_votes / total_votes * 100.0, 3)
print(f"Correy: {Li_percent_votes}% ({Li_votes})")

# print the percent of voters to the candidate O'Tooley
OTooley_candidate_as_true = df['Candidate'] == "O'Tooley"
# Create a df with only  O'Tooley
OTooley_df = df[OTooley_candidate_as_true]
# Count the number of votes  O'Tooley received
OTooley_votes = OTooley_df['Voter ID'].count()
OTooley__percent_votes = round(OTooley_votes / total_votes * 100.0, 3)
print(f"Correy: {OTooley__percent_votes}% ({OTooley_votes})")
print("-------------------------")

# Determine the winner of the popular vote by using a dictionary method
popular_vote_dict = {"Khan" : khan_votes, "Correy" : Correy_votes, "Li" : Li_votes, "O'Tooley" : OTooley_votes}
max_key = max(popular_vote_dict, key = lambda k: popular_vote_dict[k])
print(f"Winner: {max_key}")
print("-------------------------")

df.to_csv('pypoll_result.txt', header = None, sep = ' ', mode = 'a')


from pathlib import Path
import re
from axi_renamer.renamer_starter import Renamer, code_rename_map
from axi_renamer.constant_maps import pkg_map_for_folders
import os

class FolderHandler:
    def __init__(self, in_folder, out_folder, rename=False, valid_suffixes:list=None, process_files:list=None,
                       delete_sourcefile=False, target='CODE'):
        self.renamer = Renamer(target=target)
        self.root_folder = in_folder
        self.root_out_folder = out_folder
        self.valid_suffixes = valid_suffixes
        self.process_files = process_files
        self.delete_sourcefile = delete_sourcefile
        if rename:
            if delete_sourcefile:
                print('Rename & remove mode')
            else:
                print('Rename & copy mode')

        elif delete_sourcefile:
            print('Move mode')
        else:
            print('Copy mode')

    def simple_replace(self):
        self.replace_in_file_tree(self.root_folder)

    def make_renamed_dir(self, current_folder):
        relative_path = Path(current_folder).relative_to(self.root_folder)
        relative_path_str = str(relative_path)
        for row in code_rename_map:
            relative_path_str = re.sub(row[0], row[1], relative_path_str)
        relative_path_str = relative_path_str.replace('\\', '/')
        for row in pkg_map_for_folders:
            relative_path_str = re.sub(row[0], row[1], relative_path_str)
        os.path.normpath(relative_path_str)
        out_path = os.path.join(self.root_out_folder, relative_path_str)
        if not os.path.exists(out_path):
            os.makedirs(out_path)
        return out_path

    def need_to_process_file(self, file_fullname):
        if self.valid_suffixes and os.path.splitext(file_fullname)[1] not in self.valid_suffixes:
            return False

        if not self.process_files:
            return True
        b = False
        for process_file in self.process_files:
            try:
                file_fullname.index(process_file)
                b = True
                break
            except:
                continue
        return b

    def replace_in_file_tree(self, current_folder):
        files = os.listdir(current_folder)

        for file in files:
            full_name = os.path.join(current_folder, file)
            if os.path.isfile(full_name):

                if self.need_to_process_file(full_name):
                    print('Process File: ' + full_name)
                    infile = open(full_name, 'rt', encoding='UTF-8')
                    data = infile.read()
                    new_filename = str(file)

                    #Переименование файла
                    new_filename = self.renamer.get_axi_val(new_filename)

                    # Переименование других вхождений в тексте кода
                    data = self.renamer.get_axi_val(data, content_type='code')

                    # Создание переименованной директории
                    out_folder = self.make_renamed_dir(current_folder)

                    # Запись переименованнного файла
                    outfile = open(os.path.join(out_folder, new_filename), 'wt', encoding='UTF-8')
                    outfile.write(data)
                    infile.close()
                    outfile.close()

                    # Удаление исходного файла при включенной опции удаления (в т.ч. для удаления неиспользуемых файлов)
                    if self.delete_sourcefile:
                        os.remove(full_name)
            else:
                self.replace_in_file_tree(full_name)

    def free_find(self, reg_list):
        self.free_find_in_file_tree(reg_list, self.root_folder)

    def free_find_in_file_tree(self, reg_list, current_folder):
        files = os.listdir(current_folder)

        for file in files:
            full_name = os.path.join(current_folder, file)
            if os.path.isfile(full_name):

                if self.need_to_process_file(full_name):

                    infile = open(full_name, 'rt', encoding='UTF-8')
                    data = infile.read()
                    for reg in reg_list:
                        occurences = re.findall(reg, data, flags=re.IGNORECASE)
                        if bool(occurences):
                            occurences = list(set(occurences))
                            print('File: ' + full_name)
                            print(occurences)
            else:
                self.free_find_in_file_tree(reg_list, full_name)


# coding=gbk
import json
import re

FileName = ""
ethernet = []
hw_id = []

with open('config.boot', 'r') as config:
    for line in config:
        #print(line)
        if(re.match('^[a-zA-Z].*', line)):
            FileName = re.match('^[a-zA-Z].*|^}', line).group().split()[0]
            print(FileName)
        elif(re.search('.*{', line)):
            key = re.search('.*{', line).group().split()[1]
            print(key)
            ethernet.append(key)
            if(key == "lo"):
                ethernet.pop()
        elif (re.search('.*}', line)):
            continue
        else:
            val = line.split()[1]
            print(val)
            hw_id.append(val)

result = []
for i in range(len(ethernet)):
    dic = {'ethernet':ethernet[i],'hw_id':hw_id[i]}
    result.append(dic)

print("###"+FileName+"###")
newFile = FileName
with open(newFile, 'w') as name:
    for i in range(len(result)):
        jsonData = json.dumps(result[i])
        name.write(jsonData+','+'\n')


import os
import json
import sys

__config_file_path = './config.json'
__config = {}

def get_configuration():
    # Get configuration file
    if os.path.isfile(__config_file_path):
        config_file = open(__config_file_path, 'r')
        __config.update(json.loads(config_file.read()))
        config_file.close()
    else:
        print("Configuration file does not exist.")
        sys.exit(0)

def main():

    get_configuration()

    count = 0

    hosts = os.listdir(__config['out_dir_path'] + __config['hosts_dir_path'])

    for host in hosts:
        info_json = {}
        info_file_path = __config['out_dir_path'] + __config['hosts_dir_path'] + host + '/' +  __config['host_json_filename']
        if os.path.isfile(info_file_path):
            info_file = open(info_file_path, 'r')
            info_json.update(json.loads(info_file.read()))
            info_file.close()
        else:
            print('ERROR. Info file does not exist. (' + info_file_path + ')')

        try:
            gsb_dic = info_json['GSB']
            if len(gsb_dic) > 0:
                count += 1
        except:
            continue

    print('\nGSB tagged {0} hosts of {1}\n'.format(count, len(hosts)))

if __name__ == '__main__':
    main()


import pygame, random, sys, os

# the game will have meteors falling from the sky
# if player gets hit they louse
# score will be decided based on the dificlty and meteors that fall off the screen
#more psudocode
#https://docs.google.com/document/d/1jn9uq9lo-wiA5ExyEtRhGiUVOSR0DulT8XnxGLtrCSE/edit#heading=h.gakfm6o84chp

pygame.init()
#imports
walkRight = [pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\WalkRight1.png"), pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\WalkRight2.png"), pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\WalkRight3.png"), pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\WalkRight4.png"), pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\WalkRight5.png"), pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\WalkRight6.png"), pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\WalkRight7.png"), pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\WalkRight8.png")]
walkLeft = [pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\WalkLeft1.png"), pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\WalkLeft2.png"), pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\WalkLeft3.png"), pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\WalkLeft4.png"), pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\WalkLeft5.png"), pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\WalkLeft6.png"), pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\WalkLeft7.png"), pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\WalkLeft8.png")]
jumpRight = pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\JumpRight.png")
jumpLeft = pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\JumpLeft.png")
background = pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\Background.jpg")
characterRight = pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\StandingRight.png")
characterLeft = pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\StandingLeft.png")
meteor = pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\fireball.png")
Brick = pygame.image.load("C:\\Users\\walkerd24\\github\\GameDesign\\Final\\sprites\\Brick.jpg")
#screen vars
WIDTH = 1000 # screen width
HEIGHT = 800 # screen height
screen = pygame.display.set_mode((WIDTH,HEIGHT))

# control/default vars for player
running = True
click = False
alive = True
player_x = 368 #where character starts disance from left wall
player_y = 688 #where character starts distance from top
player_Size = 64 #width and height of character

#meteor code
meteor_Size = 64
meteor_pos = [random.randint(0,WIDTH-meteor_Size), 0]
meteor_list = [meteor_pos]
meteor_speed = 0

#jump code/walk
jump = False
high = 10 #height of jump
speed = 15 #needs to be able to be devided by 500 cleanly or will mess up on far side

# to control the frames
clock = pygame.time.Clock()

#control left and right move
left = False
right = False
true = False

# set up fonts
LETTER_FONT = pygame.font.SysFont('comicsans', 40)
WORD_FONT = pygame.font.SysFont('comicsans', 60)
TITLE_FONT = pygame.font.SysFont('comicsans', 70)

#control list
walkCount = 0
SpriteFrames = 24 # number of pictures for each movement times 3
Left = False #sets default side facing the right

#dificulty
dificlty = None
dificlty_multiplyer = 1
score = 0

#Scores and text input vars
scoreboard = []
base_font = pygame.font.Font(None,24)
user_text = ''
scoreboardLength = 0
namein = ''
name = [namein]

#counter for scorebord print
x=0

#text box fro player to input name for scoreboard
def nameinput():
    global user_text
    nameInputBool = True
    while nameInputBool == True:
        screen.fill((0,0,0))
        #quit
        for event in pygame.event.get():
            if event.type == pygame.QUIT:
                pygame.quit()
                sys.exit()
            #typing
            if event.type == pygame.KEYDOWN:
                if event.key == pygame.K_BACKSPACE:
                    user_text = user_text[:-1]
                #if return finish otherwise add the keypress to the text
                else:
                    if event.key == pygame.K_RETURN:
                        nameInputBool = False
                    else:
                        user_text += event.unicode
        #text box
        input_rect = pygame.Rect(30,30,140,32)
        pygame.draw.rect(screen,(255,255,255),input_rect,2)

        #basic text
        title_surface = base_font.render("enter username for scorebord. press enter to continue.",True,(255,255,255))
        screen.blit(title_surface, (input_rect.x, 10))

        #rendering text
        text_surface = base_font.render(user_text,True,(255,255,255))
        screen.blit(text_surface, (input_rect.x + 5, input_rect.y +5))

        input_rect.w = max(100, text_surface.get_width() + 10)

        pygame.display.update()

    namein = text_surface
    name[scoreboardLength] = namein

#adds to scoreboard array
def scoreboardAdd():
    global scoreboardLength
    #adds to the length of array and adds to
    nameinput()
    scoreboard.append(score)
    scoreboardLength += 1
    menu()

#level SPEED
def level_speed(score,meteor_speed):
    #changes speed depending no your score and your multiplyer
    if score < 20:
        meteor_speed = 5 * dificlty_multiplyer
    elif score < 40:
        meteor_speed = 8 * dificlty_multiplyer
    elif score < 60:
        meteor_speed = 12 * dificlty_multiplyer
    else:
        meteor_speed = 15 * dificlty_multiplyer
    return meteor_speed

#drop meteors
def meteor_fall(meteor_list):
    delay = random.random()
    #list of meteors  amount on screen delay between spawn higher the faster
    if len(meteor_list) < 30 and delay < .25:
        x_pos = random.randint(0,WIDTH-meteor_Size)
        y_pos = 0
        meteor_list.append([x_pos, y_pos])

#draw meteors
def metero_draw(meteor_list):
    #draws meteor
    for meteor_pos in meteor_list:
        screen.blit(meteor, (meteor_pos[0], meteor_pos[1]))

#updates meteor/adds score
def meteor_up(meteor_list, score):
	for xyz, meteor_pos in enumerate(meteor_list):
        #moves meteor down
		if meteor_pos[1] >= 0 and meteor_pos[1] < HEIGHT:
			meteor_pos[1] += meteor_speed
        #adds score if meteor goes of screen
		else:
			meteor_list.pop(xyz)
			score += 1
	return score

#checks for player colision wih meteor
def collision_check(player_x, player_y, meteor_pos):
    #check for colision
	for meteor_pos in meteor_list:
		if colision(player_x, player_y, meteor_pos):
			return True
	return False

#more colision
def colision(player_x, player_y, meteor_pos):
    p_x = player_x
    p_y = player_y

    m_x = meteor_pos[0]
    m_y = meteor_pos[1]
    #collision equasion
    if (m_x >= p_x and m_x < (p_x + player_Size)) or (p_x >= m_x and p_x < (m_x+meteor_Size)):
        if (m_y >= p_y and m_y < (p_y + player_Size)) or (p_y >= m_y and p_y < (m_y+meteor_Size)):
            return True
    return False

#message code
def display_message(message):
    pygame.time.delay(1000)
    win.fill((255,255,255))
    text = WORD_FONT.render(message, 1, (0,0,0))
    win.blit(text, (WIDTH/2 - text.get_width()/2, HEIGHT/2 - text.get_height()/2))
    pygame.display.update()
    pygame.time.delay(3000)

#character draw
def player_draw():
    KeyPress=pygame.key.get_pressed()
    global walkCount
    global player_x
    global speed
    global true
    global jump
    global jumpRight
    global jumpLeft
    global Left

    #sprite movement rendering code
    if walkCount + 1 >= SpriteFrames:
        walkCount = 0
    if left:
        Left = True
        if jump == True:
            screen.blit((jumpLeft), (player_x,player_y))
        else:
            screen.blit(walkLeft[walkCount//3],(player_x,player_y))
            walkCount += 1
    elif right:
        Left = False
        if jump == True:
            screen.blit((jumpRight), (player_x,player_y))
        else:
            screen.blit(walkRight[walkCount//3], (player_x,player_y))
            walkCount +=1
    else:
        if Left == True:
            screen.blit((characterLeft),(player_x,player_y))
            walkCount = 0
        else:
            screen.blit((characterRight),(player_x,player_y))
            walkCount = 0
    #screen update
    pygame.display.update()

#menu
def menu():
    global words
    global word
    global mx
    global my
    global run
    global click
    global dificlty_multiplyer
    global x

    loop = True
    while loop == True:
        #mouse position and white fill
        mx, my = pygame.mouse.get_pos()
        screen.fill((255,255,255))

        #buttons
        button_1 = pygame.Rect(105,295,200,50)
        button_2 = pygame.Rect(395,295,200,50)
        button_3 = pygame.Rect(695,295,200,50)
        button_4 = pygame.Rect(195,495,275,50)
        button_5 = pygame.Rect(565,495,200,50)
        pygame.draw.rect(screen, (255,0 ,0), button_1)
        pygame.draw.rect(screen, (255,0 ,0), button_2)
        pygame.draw.rect(screen, (255,0 ,0), button_3)
        pygame.draw.rect(screen, (255,0 ,0), button_4)
        pygame.draw.rect(screen, (255,0 ,0), button_5)

        #menu text
        text1 = TITLE_FONT.render("The Sky Is Falling", 1, (0,0,0))
        text2 = WORD_FONT.render("instructions", 1, (0,0,0))
        text2_0 = WORD_FONT.render("meteors will be falling from the sky", 1, (0,0,0))
        text2_1 = WORD_FONT.render("if you get hit you louse", 1, (0,0,0))
        text2_2 = WORD_FONT.render("use the left and right arow to move", 1, (0,0,0))
        text3 = WORD_FONT.render("easy", 1, (0,0,0))
        text4 = WORD_FONT.render("medium", 1, (0,0,0))
        text5 = WORD_FONT.render("hard", 1, (0,0,0))
        text6 = WORD_FONT.render("scoreboard", 1, (0,0,0))
        text7 = WORD_FONT.render("quit", 1, (0,0,0))

        #text display
        screen.blit(text1, (int(WIDTH/2 - (text1.get_width()/2)), 20))
        screen.blit(text2, (int(WIDTH/2 - text2.get_width()/2), 90))
        screen.blit(text2_0, (int(WIDTH/2 - text2_0.get_width()/2), 140))
        screen.blit(text2_1, (int(WIDTH/2 - text2_1.get_width()/2), 190))
        screen.blit(text2_2, (int(WIDTH/2 - text2_2.get_width()/2), 240))
        screen.blit (text3, (int(WIDTH/5 - text3.get_width()/2), 300))
        screen.blit(text4, (int(WIDTH/2 - text4.get_width()/2), 300))
        screen.blit(text5, (int(WIDTH/1.25 - text5.get_width()/2), 300))
        screen.blit(text6, (int(WIDTH/3 - text6.get_width()/2), 500))
        screen.blit(text7, (int(WIDTH/1.5 - text7.get_width()/2), 500))
        pygame.display.update()
        for event in pygame.event.get():
            #exit
            if event.type == pygame.QUIT:
                running = False
                sys.exit()
                break
            #if mouse left clicks
            if event.type == pygame.MOUSEBUTTONDOWN:
                if event.button == 1:
                    click = True
        #button colision and action
        #easy
        if button_1.collidepoint((mx,my)):
            if click:
                screen.fill((0,0,0))
                score = 0
                dificlty = "easy"
                dificlty_multiplyer = 0.75
                pygame.display.update()
                loop = False
                main()
                pass
        #normal
        if button_2.collidepoint((mx,my)):
            if click:
                screen.fill((0,0,0))
                score = 0
                dificlty = "normal"
                dificlty_multiplyer = 1
                pygame.display.update()
                loop = False
                main()
                pass
        #Hard
        if button_3.collidepoint((mx,my)):
            if click:
                screen.fill((0,0,0))
                score = 0
                dificlty = "hard"
                dificlty_multiplyer = 2
                pygame.display.update()
                loop = False
                main()
                pass
        #scoreboard
        if button_4.collidepoint((mx,my)):
            if click:
                for x in range (0,scoreboardLength):
                    nameprint = name[x]
                    scoreprint = scoreboard[x]
                    print (str(nameprint) + " " + str(scoreprint))
                x=0
                menu()
                break
        #quit
        if button_5.collidepoint((mx,my)):
            if click:
                running = False
                break
        click = False

#main
def main():
    global running
    global player_x
    global player_y
    global jump
    global high
    global speed
    global Clock
    global right
    global left
    global alive
    global score
    global meteor_speed
    while running and alive:
        clock.tick(SpriteFrames)
        #exit
        for event in pygame.event.get():
            if event.type == pygame.QUIT:
                running = False
                sys.exit()
                break
        # movement
        KeyPress=pygame.key.get_pressed()
        #check what key was get_pressed
        if KeyPress[pygame.K_LEFT] and player_x > speed: #subtract from x left movement
            player_x -= speed
            left = True
            right = False
        elif KeyPress[pygame.K_RIGHT] and player_x < WIDTH - speed: #Add to x right movement
            player_x += speed
            left = False
            right = True
        #reset code when not moving
        else:
            left = False
            right = False
            walkCount = 0
        #jump code
        if not(jump): # moving y without jump
            if KeyPress[pygame.K_SPACE]:
                jump = True
                left = False
                right = False
                walkCount = 0
        else:
            if high >=-10:
                player_y -= (high*abs(high)) /2
                high -= 1
            else:
                high = 10
                jump = False
        screen.blit(background, (0,0))

        #meteor
        meteor_fall(meteor_list)
        score = meteor_up(meteor_list, score)
        meteor_speed = level_speed(score, meteor_speed)

        #score text
        DisplayScore = WORD_FONT.render("score: " + str(score), 1, (255,255,255))
        #display text
        screen.blit(DisplayScore, (WIDTH-200, HEIGHT-40))

        #checks for colision and adds to scoreboard
        if collision_check(player_x, player_y, meteor_list) == True:
            alive = False
            scoreboardAdd()
            menu()
            break

        metero_draw(meteor_list)
        player_draw()

        clock.tick(30)
        pygame.display.update()

menu()
pygame.quit()


#!/usr/bin/python
# -*- coding:utf-8 -*-
# 2018/5/3

from rest_framework.negotiation import DefaultContentNegotiation

class CustomNegotiation(DefaultContentNegotiation):
    def get_accept_list(self, request):
        header = request.META.get('HTTP_ACCEPT', '*/*')
        nsfocus_header = [item.strip() for item in header.split(";")]
        if nsfocus_header and nsfocus_header[0].lower() == "application/nsfocus.nti.spec+json":
            header = "*/*"
        return [token.strip() for token in header.split(',')]


import os
import sys
from PIL import Image
# os is inbuilt module in python
# It is used to perform many tasks
currDir = sys.argv[1]
newDir = sys.argv[2]

allFiles = os.listdir(os.getcwd())
newDirPath = os.getcwd()+"/" + newDir
currPath = os.getcwd()+"/"+currDir
if currDir not in allFiles:
    raise Exception(currDir + " does not exists")

if newDir not in allFiles:
    os.mkdir(newDirPath)

for file_ in os.listdir(currPath):
    if file_.endswith('.jpg'):
        img = Image.open(currPath+"/"+file_)
        file_ = file_.replace('.jpg', '.png')
        img.save(newDirPath+"/"+file_, 'png')


"""This module contains the detection code for potentially insecure low-level
calls."""

from mythril.analysis import solver
from mythril.analysis.potential_issues import (
    PotentialIssue,
    get_potential_issues_annotation,
)
from mythril.laser.ethereum.state.constraints import Constraints
from mythril.laser.ethereum.transaction.symbolic import ACTORS
from mythril.analysis.module.base import DetectionModule, EntryPoint
from mythril.laser.smt import UGT, symbol_factory, Or, BitVec
from mythril.laser.ethereum.natives import PRECOMPILE_COUNT
from mythril.laser.ethereum.state.global_state import GlobalState
from mythril.exceptions import UnsatError
import logging

from mythril.laser.smt import simplify
from copy import deepcopy
from mythril.analysis.report import Issue
from mythril.analysis.issue_annotation import IssueAnnotation
from mythril.laser.ethereum.util import get_concrete_int
from mythril.laser.smt.bool import And
import re

TRANSACTION_ORDER_DEPENDENCY = 144

log = logging.getLogger(__name__)

DESCRIPTION = '''
MODULE DESCRIPTION:
This module finds the existance of transaction order dependence vulnerabilities.
The following webpage contains an extensive description of the vulnerability:
https://consensys.github.io/smart-contract-best-practices/known_attacks/#transaction-ordering-dependence-tod-front-running
'''

class TransactionOrderDependency(DetectionModule):
    """This module searches for low level calls (e.g. call.value()) that
    forward all gas to the callee."""

    name = "Transaction Order Dependency"
    swc_id = TRANSACTION_ORDER_DEPENDENCY
    description = DESCRIPTION
    entry_point = EntryPoint.CALLBACK
    pre_hooks = ["CALL"]

    def _execute(self, state: GlobalState) -> None:
        """

        :param state:
        :return:
        """

        potential_issues = self._analyze_state(state)

        annotation = get_potential_issues_annotation(state)
        annotation.potential_issues.extend(potential_issues)

    def _analyze_state(self, state: GlobalState):
        """

        :param state:
        :return:
        """
        try:

            interesting_storages = list(_get_influencing_storages(state))
            changing_sstores = list(_get_influencing_sstores(state, interesting_storages))

            if len(changing_sstores) > 0:

                to = state.mstate.stack[-2]

                constraints = (
                    state.world_state.constraints
                    + [to == ACTORS.attacker]
                )

                transaction_sequence = solver.get_transaction_sequence(
                    state, constraints
                )

                instruction = state.get_current_instruction()
                issue = Issue(
                    contract=state.environment.active_account.contract_name,
                    function_name=state.environment.active_function_name,
                    address=instruction["address"],
                    swc_id=TRANSACTION_ORDER_DEPENDENCY,
                    bytecode=state.environment.code.bytecode,
                    title="Unprotected Transaction Order",
                    severity="High",
                    description_head="description_head",
                    description_tail="description_tail",
                    transaction_sequence=transaction_sequence,
                    gas_used=(state.mstate.min_gas_used, state.mstate.max_gas_used),
                )

                state.annotate(
                    IssueAnnotation(
                        conditions=[And(*constraints)], issue=issue, detector=self
                    )
                )

                return [issue]
        except:
            log.debug("No model found")
        return []

def _get_influencing_storages(state: GlobalState):
    gas = state.mstate.stack[-1]
    to = state.mstate.stack[-2]
    value = state.mstate.stack[-3]
    storages = []

    if to.symbolic is False:
        storages += _dependent_on_storage(to.value)
    if value.symbolic is False:
        storages += _dependent_on_storage(value.value)

    for storage in storages:
        variable = _get_storage_variable(storage, state)
        can_change = _can_change(state.world_state.constraints, variable, gas)
        if can_change:
            yield storage

def _dependent_on_storage(expression):

    pattern = re.compile(r"storage_[a-z0-9_&^]+")
    return pattern.findall(str(simplify(expression)))

def _get_storage_variable(storage, state):

    index = int(re.search('[0-9]+', storage).group())

    try:
        return state.environment.active_account.storage(index)
    except:
        return None

def _can_change(constraints, variable, gas):
    _constraints = deepcopy(constraints)

    try:
        model = solver.get_model(_constraints + [UGT(gas, symbol_factory.BitVecVal(2300, 256))])

        try:
            initial_value = int(str(model.eval(variable, model_completion=True)))
            _constraints.append(variable != initial_value)
            solver.get_model(_constraints)
            return True

        except UnsatError:
            return False

    except UnsatError:
        return False

def _get_influencing_sstores(state, interesting_storages):
    for sstore_state, node in _get_states_with_opcode(state, "SSTORE"):
        index, value = sstore_state.mstate.stack[-1], sstore_state.mstate.stack[-2]
        try:
            index = get_concrete_int(index)
        except AttributeError:
            index = str(index)
        if "storage_{}".format(index) not in interesting_storages:
            continue
        yield sstore_state, node

# sstore가 들어가 잇는 state가져오기
# 예전과 방식이 바뀐 느낌.. 가져오느게 다름.
def _get_states_with_opcode(state, opcode):
    for k in state.node:
        node = state.nodes[k]
        for _state in node.states:
            if state.get_current_instruction()["opcode"] == opcode:
                yield state, node

detector = TransactionOrderDependency()


sequence = '3113322113' + '_'

# Part 1 & 2: find the length of the sequence after 40 and 50 steps
# Approach:
# At the start of each step start by initializing a new sequence ''
# Loop over current sequnce and keep track amount of repeated numbers
# If next number != previous number add the amount of numbers to the new sequence and then the previous number
# Rinse and repeat
# Note: To make the code cleaner for the last number, a non-digit character is added at the end of the aequence
# by doing this the last number will always be accounted for in the next sequence while the character is skipped

for step in range(50):
    next_sequence = ''
    current_number = sequence[0]
    number_count = 0
    for number in sequence:
        if number == current_number:
            number_count += 1
        else:
            next_sequence += str(number_count) + current_number
            current_number = number
            number_count = 1
    if step == 39:
        print(f'Part 1: The length after 40 steps = {len(next_sequence)}')
    if step == 49:
        print(f'Part 2: The length after 50 steps = {len(next_sequence)}')
    # print(next_sequence)
    sequence = next_sequence + '_'


infile=open('menu.txt','r')
outfile=open("outt.txt",'w')
menu=[]
for line in infile:
    line=line.strip()
    items=line.split()
    for i in range(len(items)):
        items[i]=items[i].strip()
        items[i]=items[i].strip(".:$")
        if i==len(items)-1:
            items[i]=float(items[i])
    items.remove("")
    menu.append(items)

count=(len(menu))*[0]
user=input("=")
while user:
    for i in range(len(menu)):
        if user==menu[i][0]:
            count[i]=count[i]+1
    user=input("=")
print(count)

price=(len(menu))*[0]
for i in range(len(menu)):
    price[i]=count[i]*menu[i][2]

for i in range(len(menu)):
    print("%5s. %5s : %5s * %1s = %5s"%(menu[i][0],menu[i][1],menu[i][2],count[i],price[i]),file=outfile)